- The paper presents a novel framework (OLSF-TRS) that integrates bisimulation-based abstraction with Transformer-based neural combinatorial optimization and MAPPO for efficient tote-handling order fulfillment.
- The methodology decomposes complex warehouse decisions into order assignment, tote matching, and robot scheduling, achieving near-optimal performance with millisecond-level inference.
- Experimental results demonstrate up to 14.9% reduction in tote movements and improved system stability across varied warehouse scales and architectures.
Omni-scale Learning-Based Sequential Decision Framework for Order Fulfillment of Tote-Handling Robotic Systems
Context and Motivation
Operational dynamics in automated e-commerce and intralogistics facilities have evolved considerably due to the shift from palletized to tote-based handling, driven by the demand for high-frequency, small-batch fulfillment. As tote-handling robotic systems (THRS) become the industry standard—evident in deployments like Hairobotics and Exotec—prevailing decision frameworks have remained system-specific, non-transferable, and limited in scalability and generalization. Traditional approaches, bound to rule-based heuristics or combinatorial optimizers, fail to address the hierarchical, tightly coupled nature and computational complexity of warehouse fulfillment at the order, tote, and robot layers. The presented work introduces the Omni-scale Learning-based Sequential Decision Framework for Order Fulfillment of Tote-handling Robotic Systems (OLSF-TRS), designed to deliver a generalized, scalable, and modular paradigm that marries structured neural combinatorial optimization (NCO) with multi-agent reinforcement learning (MARL).
Methodological Framework
OLSF-TRS formulates the sequential decision process as a hierarchical pipeline spanning order assignment, tote matching, and robot scheduling, each as a Markov Decision Process (MDP) with large, correlated state and action spaces. To address the curse of dimensionality, the authors employ bisimulation quotienting (BQ) to construct abstract MDPs (BQ-MDPs), where behaviorally equivalent states yielding identical transition distributions and rewards are merged. This abstraction is structurally preserved regardless of system heterogeneity (e.g., 2D Multi-Tote vs. 3D Rack-Climbing robots) and facilitates learning policies that generalize across warehouse scales and architectures.
Neural Combinatorial Optimization Policy Heads
Each subproblem within the OLSF-TRS pipeline (order assignment, tote matching, robot scheduling) is solved via dedicated BQ-NCO policy heads. These policy heads are instantiated as Transformer-based models pretrained through imitation learning using exact solvers (e.g., Gurobi) on small-scale warehouse instances, ensuring accurate local policies. Modular design allows policies to transfer across problems, instances, and architectures.
Multi-Agent Proximal Policy Optimization (MAPPO) Coordination
For large-scale, stochastic warehouse environments, OLSF-TRS integrates the pretrained policy heads within a MAPPO-based architecture. Order, Tote, and Robot agents operate as decentralized actors with centralized training via a global critic. This enables robust, coordinated negotiation of complex, cross-agent dependencies. The global reward, reflecting overall system efficiency measured by the total number of tote movements, aligns all agents towards throughput-optimal and cost-effective policies. The centralized critic utilizes Generalized Advantage Estimation (GAE), improving credit assignment and mitigating policy degradation seen in decentralized MARL methods.
Experimental Analysis
Evaluation Setup
Eighteen synthetic benchmark instances emulate real-world THRS across scales from small (S-1 to S-9, up to 60 SKUs and 20 orders) to large (L-1 to L-15, up to 600 SKUs and 350 orders), covering both low- and high-concurrency scenarios and two principal architectures (2D Multi-Tote and 3D Rack-Climbing robots). Baseline comparisons span state-of-the-art heuristics (Collaborative SKU-Group Heuristic (C-SGH)), logic-centric methods (Robot-first, Group-based Greedy), and independent MARL (IPPO).
On small-scale instances, OLSF-TRS achieves agent-level solution quality within 3.5% optimality gap relative to exact solvers, with millisecond-level inference times, establishing the reliability and efficiency of local decisions.
For large-scale scenarios, OLSF-TRS demonstrates consistent superiority over all baselines:
- Hairobotic (2D Multi-Tote) system: OLSF-TRS reduces tote movements by up to 14.9% over the best heuristic (C-SGH) and over 19% over IPPO, with more pronounced gains as concurrency increases.
- Exotec (3D Rack-Climbing) system: Analogous cost reductions are witnessed (over 43% relative to rule-based and 25% over group-based greedy heuristics in the largest instances).
Crucially, as robot-to-order ratios decline and resource contention intensifies (L-10 to L-15), heuristic and decentralized learning baselines experience significant performance collapse, whereas OLSF-TRS maintains near-linear scaling in operational cost and millisecond-scale policy inference. These improvements manifest as increased throughput, decreased energy consumption, and enhanced stability, underpinning OLSF-TRS's applicability to real-time, high-density industrial environments.
Theoretical and Practical Implications
The central contribution of this work is the demonstration that high-dimensional, tightly coupled sequential warehouse decision-making can be effectively addressed without recourse to monolithic joint optimization. Hierarchical decomposition with task-specific neural policies, together with a centralized coordination mechanism, circumvents both the intractability of the full joint problem and the suboptimality of fragmented heuristics.
The bisimulation-based abstraction confirms empirical viability beyond its theoretical elegance, enabling substantial compression of state space and dramatically improved data efficiency in policy training. The integration of BQ-NCO and MAPPO resolves conventional MARL issues such as reward confusion (credit assignment problem) and congestion-induced inefficiency, yielding robust inter-agent coordination under heavy system load.
From an industrial standpoint, the framework's modularity supports rapid reconfiguration across warehouse layouts and robotic platforms, offering a unified foundation for ongoing automation, scenario simulation, and adaptive control.
Limitations and Future Directions
Certain simplifications persist: homogeneous robot assumptions, abstracted low-level motion planning, and the omission of explicit physical constraints such as battery management. Adapting BQ-MDP construction for unstructured or dynamic environments, integrating explicit path planning, and accommodating heterogeneous fleets are critical next steps. The framework's architecture is amenable to enhancements with real-time sensing, online adaptation, and the inclusion of high-level strategic models (e.g., demand forecasting via large foundation models).
Conclusion
OLSF-TRS represents a significant advancement in the integration of learning-based combinatorial optimization and cooperative MARL for large-scale warehouse order-fulfillment. The methodology ensures scalable, transferable, and robust decision-making across varied automated intralogistics systems, achieving strong numerical advantages in both cost and operational efficiency relative to domain baselines. The framework establishes a basis for continued research toward self-optimizing, adaptive robotic fulfillment centers supporting the future of automation in logistics and e-commerce.
Reference:
"Omni-scale Learning-based Sequential Decision Framework for Order Fulfillment of Tote-handling Robotic Systems" (2605.08758)
