Scalable Multi-agent Reinforcement Learning for Factory-wide Dynamic Scheduling (2409.13571v1)

Abstract: Real-time dynamic scheduling is a crucial but notoriously challenging task in modern manufacturing processes due to its high decision complexity. Recently, reinforcement learning (RL) has been gaining attention as an impactful technique to handle this challenge. However, classical RL methods typically rely on human-made dispatching rules, which are not suitable for large-scale factory-wide scheduling. To bridge this gap, this paper applies a leader-follower multi-agent RL (MARL) concept to obtain desired coordination after decomposing the scheduling problem into a set of sub-problems that are handled by each individual agent for scalability. We further strengthen the procedure by proposing a rule-based conversion algorithm to prevent catastrophic loss of production capacity due to an agent's error. Our experimental results demonstrate that the proposed model outperforms the state-of-the-art deep RL-based scheduling models in various aspects. Additionally, the proposed model provides the most robust scheduling performance to demand changes. Overall, the proposed MARL-based scheduling model presents a promising solution to the real-time scheduling problem, with potential applications in various manufacturing industries.

• The paper proposes a leader-follower multi-agent reinforcement learning model that decomposes complex scheduling into manageable sub-problems.
• It eliminates reliance on dispatching rules by learning adaptive policies, significantly reducing tardiness and maximizing completion rate.
• Incorporating a rule-based conversion algorithm, the model ensures robust decision-making and scalability under varying demand levels.

Scalable Multi-agent Reinforcement Learning for Factory-wide Dynamic Scheduling

The paper "Scalable Multi-agent Reinforcement Learning for Factory-wide Dynamic Scheduling" addresses the inherently complex and dynamic nature of scheduling in semiconductor manufacturing. Such environments are characterized by fluctuating demands, high levels of operational constraints, and significant variability, necessitating robust scheduling methodologies.

The authors propose a multi-agent reinforcement learning (MARL) model that leverages a leader-follower framework to decompose a large-scale scheduling problem into more manageable sub-problems. This approach enhances scalability while ensuring tight coordination among agents to achieve global optimization goals. The followers, each responsible for specific operations, are guided by abstract goal vectors generated by a leader agent, thus fostering an effective hierarchical decision-making process.

Key Contributions and Methodology

1. RL-based Scheduling Model Without Dispatching Rules: The model eschews traditional reliance on human-designed dispatching rules. Instead, it learns policies through reinforcement learning, thereby adapting more effectively to the stochastic nature of factory environments.
2. Leader-Follower MARL Concept: The paper introduces a leader-follower MARL model. The leader coordinates among followers by distributing abstract goal vectors at the beginning of each shift. This approach mitigates the challenges of large joint action spaces and intricate inter-agent dependencies, a notable advancement over standard DRL methods.
3. Rule-based Conversion Algorithm: To prevent significant production losses due to erroneous agent decisions, a rule-based conversion algorithm is integrated. This algorithm overrides follower decisions when they pose a substantial risk, thus enhancing overall robustness.

Experimental Results

The model is rigorously evaluated using two distinct scenarios based on real production data: short-term and long-term manufacturing environments. Various demand levels (low, medium, high) are tested to assess the model's adaptability and performance under different conditions.

• Performance Metrics: The evaluation focuses on four key metrics: tardiness, number of changeovers, cumulative idle time, and completion rate against demand. The proposed model shows superior performance across most metrics, particularly excelling in minimizing tardiness and maximizing completion rate, which are critical for maintaining high productivity.
• Completion Rate: The proposed model significantly outperforms DRL-JSSP and DRL-DFJSS in terms of completion rate. This improvement is consistent across different demand levels, underscoring the model's robustness and scalability.

Comparative Analysis

The proposed model demonstrates a marked improvement over existing RL-based scheduling methods, such as DRL-JSSP and DRL-DFJSS. The benchmark models, which rely heavily on predefined dispatching rules, exhibit limited adaptability to factory-wide scheduling challenges. In contrast, the MARL approach, particularly with the rule-based conversion mechanism, addresses critical operational constraints effectively, demonstrating the practical viability of the proposed model in real-world settings.

Implications and Future Directions

The implications of this research are significant for the manufacturing industry, particularly in contexts requiring dynamic and scalable scheduling solutions. The robustness of the MARL-based approach suggests potential applications beyond semiconductor manufacturing, including various process industries where complex scheduling is a common challenge.

Theoretically, this research contributes to the development of MARL algorithms, particularly in hierarchical decision-making frameworks. The introduction of a leader-follower model with abstract goals can inspire future studies aiming to improve coordination and scalability in multi-agent systems.

Future Developments:

• Adaptive Models: An important future direction involves developing models that dynamically evolve with changes in factory settings, reducing the need for frequent retraining.
• Extended Use Cases: Testing the proposed model in different industrial settings can further validate its versatility and adaptability.
• Enhanced Coordination Mechanisms: Future research could explore more sophisticated coordination strategies among agents to handle even more complex scheduling environments.

In summary, the proposed model represents a significant advancement in dynamic scheduling for semiconductor manufacturing. Its ability to learn effective policies without dispatching rules, coupled with a robust mechanism to prevent production losses, positions it as a promising solution for real-world manufacturing challenges.