Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
97 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Reinforcement and Deep Reinforcement Learning-based Solutions for Machine Maintenance Planning, Scheduling Policies, and Optimization (2307.03860v1)

Published 7 Jul 2023 in cs.LG and cs.AI

Abstract: Systems and machines undergo various failure modes that result in machine health degradation, so maintenance actions are required to restore them back to a state where they can perform their expected functions. Since maintenance tasks are inevitable, maintenance planning is essential to ensure the smooth operations of the production system and other industries at large. Maintenance planning is a decision-making problem that aims at developing optimum maintenance policies and plans that help reduces maintenance costs, extend asset life, maximize their availability, and ultimately ensure workplace safety. Reinforcement learning is a data-driven decision-making algorithm that has been increasingly applied to develop dynamic maintenance plans while leveraging the continuous information from condition monitoring of the system and machine states. By leveraging the condition monitoring data of systems and machines with reinforcement learning, smart maintenance planners can be developed, which is a precursor to achieving a smart factory. This paper presents a literature review on the applications of reinforcement and deep reinforcement learning for maintenance planning and optimization problems. To capture the common ideas without losing touch with the uniqueness of each publication, taxonomies used to categorize the systems were developed, and reviewed publications were highlighted, classified, and summarized based on these taxonomies. Adopted methodologies, findings, and well-defined interpretations of the reviewed studies were summarized in graphical and tabular representations to maximize the utility of the work for both researchers and practitioners. This work also highlights the research gaps, key insights from the literature, and areas for future work.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (82)
  1. W. Wang, “An overview of the recent advances in delay-time-based maintenance modelling,” Reliability Engineering & System Safety, vol. 106, pp. 165–178, 2012.
  2. Springer, 2014.
  3. C. Andriotis and K. Papakonstantinou, “Deep reinforcement learning driven inspection and maintenance planning under incomplete information and constraints,” Reliability Engineering & System Safety, vol. 212, p. 107551, 2021.
  4. H. Wang, Q. Yan, and S. Zhang, “Integrated scheduling and flexible maintenance in deteriorating multi-state single machine system using a reinforcement learning approach,” Advanced Engineering Informatics, vol. 49, p. 101339, 2021.
  5. M. Ahang, M. Jalayer, A. Shojaeinasab, O. Ogunfowora, T. Charter, and H. Najjaran, “Synthesizing rolling bearing fault samples in new conditions: A framework based on a modified cgan,” Sensors, vol. 22, no. 14, p. 5413, 2022.
  6. A. Sarkara, D. K. Beherab, and S. Kumarc, “Maintenance policies of single and multi-unit systems in the past and present,” Asian Review of Mechnical Engineering, vol. 15, 2012.
  7. J. Huang, Q. Chang, and N. Chakraborty, “Machine preventive replacement policy for serial production lines based on reinforcement learning,” in 2019 IEEE 15th International Conference on Automation Science and Engineering (CASE), pp. 523–528, IEEE, 2019.
  8. J.-H. Chiang and J. Yuan, “Optimal maintenance policy for a deteriorating production system under inspection,” Journal of the Chinese Institute of Industrial Engineers, vol. 18, no. 6, pp. 73–83, 2001.
  9. X. Wang, H. Wang, and C. Qi, “Multi-agent reinforcement learning based maintenance policy for a resource constrained flow line system,” Journal of Intelligent Manufacturing, vol. 27, no. 2, pp. 325–333, 2016.
  10. A. Shojaeinasab, T. Charter, M. Jalayer, M. Khadivi, O. Ogunfowora, N. Raiyani, M. Yaghoubi, and H. Najjaran, “Intelligent manufacturing execution systems: A systematic review,” Journal of Manufacturing Systems, vol. 62, pp. 503–522, 2022.
  11. Z. Liu and L. Zhang, “A review of failure modes, condition monitoring and fault diagnosis methods for large-scale wind turbine bearings,” Measurement, vol. 149, p. 107002, 2020.
  12. W. Li, G.-G. Wang, and A. H. Gandomi, “A survey of learning-based intelligent optimization algorithms,” Archives of Computational Methods in Engineering, vol. 28, no. 5, pp. 3781–3799, 2021.
  13. P. Bharti and S. Jain, “State-of-the-art in optimisation and heuristics to solve manufacturing scheduling problem,” International Journal of Operational Research, vol. 44, no. 3, pp. 292–348, 2022.
  14. S. Nguyen, Y. Mei, and M. Zhang, “Genetic programming for production scheduling: a survey with a unified framework,” Complex & Intelligent Systems, vol. 3, no. 1, pp. 41–66, 2017.
  15. S. Anderer, T.-H. Vu, B. Scheuermann, and S. Mostaghim, “Meta heuristics for dynamic machine scheduling: A review of research efforts and industrial requirements.,” IJCCI, pp. 192–203, 2018.
  16. M. Khadivi, T. Charter, M. Yaghoubi, M. Jalayer, M. Ahang, A. Shojaeinasab, and H. Najjaran, “Deep reinforcement learning for machine scheduling: Methodology, the state-of-the-art, and future directions,” Available at SSRN 4319327, 2022.
  17. R. S. Sutton and A. G. Barto, Reinforcement learning: An introduction. MIT press, 2018.
  18. J. Arora, Introduction to optimum design. Elsevier, 2004.
  19. J. Huang, Q. Chang, and J. Arinez, “Deep reinforcement learning based preventive maintenance policy for serial production lines,” Expert Systems with Applications, vol. 160, p. 113701, 2020.
  20. J. Su, J. Huang, S. Adams, Q. Chang, and P. A. Beling, “Deep multi-agent reinforcement learning for multi-level preventive maintenance in manufacturing systems,” Expert Systems with Applications, vol. 192, p. 116323, 2022.
  21. Y. Liu, Y. Chen, and T. Jiang, “Dynamic selective maintenance optimization for multi-state systems over a finite horizon: A deep reinforcement learning approach,” European Journal of Operational Research, vol. 283, no. 1, pp. 166–181, 2020.
  22. Q. Yan, W. Wu, and H. Wang, “Deep reinforcement learning for distributed flow shop scheduling with flexible maintenance,” Machines, vol. 10, no. 3, p. 210, 2022.
  23. X. Wang, G. Zhang, Y. Li, and N. Qu, “A heuristically accelerated reinforcement learning method for maintenance policy of an assembly line,” Journal of Industrial and Management Optimization, 2022.
  24. L. Pinciroli, P. Baraldi, G. Ballabio, M. Compare, and E. Zio, “Optimization of the operation and maintenance of renewable energy systems by deep reinforcement learning,” Renewable Energy, vol. 183, pp. 752–763, 2022.
  25. J. Wang, D. Lei, and J. Cai, “An adaptive artificial bee colony with reinforcement learning for distributed three-stage assembly scheduling with maintenance,” Applied Soft Computing, vol. 117, p. 108371, 2022.
  26. P. D. Paraschos, G. K. Koulinas, and D. E. Koulouriotis, “Reinforcement learning for combined production-maintenance and quality control of a manufacturing system with deterioration failures,” Journal of Manufacturing Systems, vol. 56, pp. 470–483, 2020.
  27. Q. Yan, H. Wang, and F. Wu, “Digital twin-enabled dynamic scheduling with preventive maintenance using a double-layer q-learning algorithm,” Computers & Operations Research, vol. 144, p. 105823, 2022.
  28. M. Compare, L. Bellani, E. Cobelli, E. Zio, F. Annunziata, F. Carlevaro, and M. Sepe, “A reinforcement learning approach to optimal part flow management for gas turbine maintenance,” Proceedings of the Institution of Mechanical Engineers, Part O: Journal of Risk and Reliability, vol. 234, no. 1, pp. 52–62, 2020.
  29. A. Medury and S. Madanat, “System-level optimization of maintenance and replacement decisions for road networks,” Applications of Statistics and Probability in Civil Engineering, pp. 235–242, 2011.
  30. Y. Chen, Y. Liu, and T. Xiahou, “A deep reinforcement learning approach to dynamic loading strategy of repairable multistate systems,” IEEE Transactions on Reliability, vol. 71, no. 1, pp. 484–499, 2021.
  31. Y. Zhou, B. Li, and T. R. Lin, “Maintenance optimisation of multicomponent systems using hierarchical coordinated reinforcement learning,” Reliability Engineering & System Safety, vol. 217, p. 108078, 2022.
  32. N. Yousefi, S. Tsianikas, and D. W. Coit, “Dynamic maintenance model for a repairable multi-component system using deep reinforcement learning,” Quality Engineering, vol. 34, no. 1, pp. 16–35, 2022.
  33. R. Lamprecht, F. Wurst, and M. F. Huber, “Reinforcement learning based condition-oriented maintenance scheduling for flow line systems,” in 2021 IEEE 19th International Conference on Industrial Informatics (INDIN), pp. 1–7, IEEE, 2021.
  34. K. Verbert, B. De Schutter, and R. Babuška, “Timely condition-based maintenance planning for multi-component systems,” Reliability Engineering & System Safety, vol. 159, pp. 310–321, 2017.
  35. N. Zhang and W. Si, “Deep reinforcement learning for condition-based maintenance planning of multi-component systems under dependent competing risks,” Reliability Engineering & System Safety, vol. 203, p. 107094, 2020.
  36. A. Kuhnle, J. Jakubik, and G. Lanza, “Reinforcement learning for opportunistic maintenance optimization,” Production Engineering, vol. 13, no. 1, pp. 33–41, 2019.
  37. L. Pinciroli, P. Baraldi, G. Ballabio, M. Compare, and E. Zio, “Deep reinforcement learning based on proximal policy optimization for the maintenance of a wind farm with multiple crews,” Energies, vol. 14, no. 20, p. 6743, 2021.
  38. J. Liu, Q. Chang, G. Xiao, and S. Biller, “The costs of downtime incidents in serial multistage manufacturing systems,” Journal of Manufacturing Science and Engineering, vol. 134, no. 2, 2012.
  39. J. Zou, Q. Chang, Y. Lei, and J. Arinez, “Production system performance identification using sensor data,” IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 48, no. 2, pp. 255–264, 2016.
  40. A. Adsule, M. Kulkarni, and A. Tewari, “Reinforcement learning for optimal policy learning in condition-based maintenance,” IET Collaborative Intelligent Manufacturing, vol. 2, no. 4, pp. 182–188, 2020.
  41. P. Zhang, X. Zhu, and M. Xie, “A model-based reinforcement learning approach for maintenance optimization of degrading systems in a large state space,” Computers & Industrial Engineering, vol. 161, p. 107622, 2021.
  42. Y. Zhao and C. Smidts, “Reinforcement learning for adaptive maintenance policy optimization under imperfect knowledge of the system degradation model and partial observability of system states,” Reliability Engineering & System Safety, vol. 224, p. 108541, 2022.
  43. Z. Mahmoodzadeh, K.-Y. Wu, E. Lopez Droguett, and A. Mosleh, “Condition-based maintenance with reinforcement learning for dry gas pipeline subject to internal corrosion,” Sensors, vol. 20, no. 19, p. 5708, 2020.
  44. S. Peng et al., “Reinforcement learning with gaussian processes for condition-based maintenance,” Computers & Industrial Engineering, vol. 158, p. 107321, 2021.
  45. J. Ribeiro, P. Andrade, M. Carvalho, C. Silva, B. Ribeiro, and L. Roque, “Playful probes for design interaction with machine learning: A tool for aircraft condition-based maintenance planning and visualisation,” Mathematics, vol. 10, no. 9, p. 1604, 2022.
  46. Z. Li, S. Zhong, and L. Lin, “An aero-engine life-cycle maintenance policy optimization algorithm: Reinforcement learning approach,” Chinese Journal of Aeronautics, vol. 32, no. 9, pp. 2133–2150, 2019.
  47. M. Knowles, D. Baglee, and S. Wermter, “Reinforcement learning for scheduling of maintenance,” in International Conference on Innovative Techniques and Applications of Artificial Intelligence, pp. 409–422, Springer, 2010.
  48. J. Lee and M. Mitici, “Deep reinforcement learning for predictive aircraft maintenance using probabilistic remaining-useful-life prognostics,” Reliability Engineering & System Safety, vol. 230, p. 108908, 2023.
  49. A. Xanthopoulos, A. Kiatipis, D. E. Koulouriotis, and S. Stieger, “Reinforcement learning-based and parametric production-maintenance control policies for a deteriorating manufacturing system,” IEEE Access, vol. 6, pp. 576–588, 2017.
  50. Y. Hu, X. Miao, J. Zhang, J. Liu, and E. Pan, “Reinforcement learning-driven maintenance strategy: A novel solution for long-term aircraft maintenance decision optimization,” Computers & Industrial Engineering, vol. 153, p. 107056, 2021.
  51. F. J. Zhao and Y. Zhou, “Wind farm maintenance scheduling using soft actor-critic deep reinforcement learning,” in 2022 Global Reliability and Prognostics and Health Management (PHM-Yantai), pp. 1–6, IEEE, 2022.
  52. H. Yang, W. Li, and B. Wang, “Joint optimization of preventive maintenance and production scheduling for multi-state production systems based on reinforcement learning,” Reliability Engineering & System Safety, vol. 214, p. 107713, 2021.
  53. K. Wesendrup and B. Hellingrath, “Post-prognostics demand management, production, spare parts and maintenance planning for a single-machine system using reinforcement learning,” Computers & Industrial Engineering, vol. 179, p. 109216, 2023.
  54. H. Rasay, F. Naderkhani, and A. M. Golmohammadi, “Reinforcement learning based on stochastic dynamic programming for condition-based maintenance of deteriorating production processes,” in 2022 IEEE International Conference on Prognostics and Health Management (ICPHM), pp. 17–24, IEEE, 2022.
  55. N. Yousefi, S. Tsianikas, and D. W. Coit, “Reinforcement learning for dynamic condition-based maintenance of a system with individually repairable components,” Quality Engineering, vol. 32, no. 3, pp. 388–408, 2020.
  56. J. Hu, Y. Wang, Y. Pang, and Y. Liu, “Optimal maintenance scheduling under uncertainties using linear programming-enhanced reinforcement learning,” Engineering Applications of Artificial Intelligence, vol. 109, p. 104655, 2022.
  57. R. Rocchetta, L. Bellani, M. Compare, E. Zio, and E. Patelli, “A reinforcement learning framework for optimal operation and maintenance of power grids,” Applied energy, vol. 241, pp. 291–301, 2019.
  58. Y. Yang and L. Yao, “Optimization method of power equipment maintenance plan decision-making based on deep reinforcement learning,” Mathematical Problems in Engineering, vol. 2021, 2021.
  59. K. S. H. Ong, W. Wang, D. Niyato, and T. Friedrichs, “Deep-reinforcement-learning-based predictive maintenance model for effective resource management in industrial iot,” IEEE Internet of Things Journal, vol. 9, no. 7, pp. 5173–5188, 2021.
  60. J. Ruan, Z. Wang, F. T. Chan, S. Patnaik, and M. K. Tiwari, “A reinforcement learning-based algorithm for the aircraft maintenance routing problem,” Expert Systems with Applications, vol. 169, p. 114399, 2021.
  61. P. Andrade, C. Silva, B. Ribeiro, and B. F. Santos, “Aircraft maintenance check scheduling using reinforcement learning,” Aerospace, vol. 8, no. 4, p. 113, 2021.
  62. Q. Wu, Q. Feng, Y. Ren, Q. Xia, Z. Wang, and B. Cai, “An intelligent preventive maintenance method based on reinforcement learning for battery energy storage systems,” IEEE Transactions on Industrial Informatics, vol. 17, no. 12, pp. 8254–8264, 2021.
  63. B. Li and Y. Zhou, “Multi-component maintenance optimization: An approach combining genetic algorithm and multiagent reinforcement learning,” in 2020 Global Reliability and Prognostics and Health Management (PHM-Shanghai), pp. 1–7, IEEE, 2020.
  64. Y. Zhang, D. Zhang, X. Zhang, L. Qiu, F. T. Chan, Z. Wang, and S. Zhang, “Guided probabilistic reinforcement learning for sampling-efficient maintenance scheduling of multi-component system,” Applied Mathematical Modelling, vol. 119, pp. 677–697, 2023.
  65. J. Chen and Y. Wang, “A deep reinforcement learning approach for maintenance planning of multi-component systems with complex structure,” Neural Computing and Applications, pp. 1–14, 2023.
  66. C. Zhang, Y.-F. Li, and D. W. Coit, “Deep reinforcement learning for dynamic opportunistic maintenance of multi-component systems with load sharing,” IEEE Transactions on Reliability, 2022.
  67. Y. Liu and X. Qian, “Selective maintenance optimization with stochastic break duration based on reinforcement learning,” Eksploatacja i Niezawodność, vol. 24, no. 4, 2022.
  68. F. Kosanoglu, M. Atmis, and H. H. Turan, “A deep reinforcement learning assisted simulated annealing algorithm for a maintenance planning problem,” Annals of Operations Research, pp. 1–32, 2022.
  69. K. S. H. Ong, W. Wang, N. Q. Hieu, D. Niyato, and T. Friedrichs, “Predictive maintenance model for iiot-based manufacturing: A transferable deep reinforcement learning approach,” IEEE Internet of Things Journal, 2022.
  70. M. L. R. Rodríguez, S. Kubler, A. de Giorgio, M. Cordy, J. Robert, and Y. Le Traon, “Multi-agent deep reinforcement learning based predictive maintenance on parallel machines,” Robotics and Computer-Integrated Manufacturing, vol. 78, p. 102406, 2022.
  71. C. Senthil and R. Sudhakara Pandian, “Proactive maintenance model using reinforcement learning algorithm in rubber industry,” Processes, vol. 10, no. 2, p. 371, 2022.
  72. P. Rokhforoz, M. Montazeri, and O. Fink, “Safe multi-agent deep reinforcement learning for joint bidding and maintenance scheduling of generation units,” Reliability Engineering & System Safety, p. 109081, 2023.
  73. J. Chatterjee and N. Dethlefs, “Deep reinforcement learning for maintenance planning of offshore vessel transfer,” in Developments in Renewable Energies Offshore, pp. 435–443, CRC Press, 2020.
  74. M. Elbasheer, F. Longo, G. Mirabelli, A. Padovano, V. Solina, and S. Talarico, “Integrated prescriptive maintenance and production planning: a machine learning approach for the development of an autonomous decision support agent,” IFAC-PapersOnLine, vol. 55, no. 10, pp. 2605–2610, 2022.
  75. M. Feng and Y. Li, “Predictive maintenance decision making based on reinforcement learning in multistage production systems,” IEEE Access, vol. 10, pp. 18910–18921, 2022.
  76. X. Wang, C. Qi, H. Wang, Q. Si, and G. Zhang, “Resilience-driven maintenance scheduling methodology for multi-agent production line system,” in The 27th Chinese Control and Decision Conference (2015 CCDC), pp. 614–619, IEEE, 2015.
  77. W. Ferreira, C. Cavalcante, and P. Do Van, “Deep reinforcement learning-based maintenance decision-making for a steel production line,” in 31st European Safety and Reliability Conference, ESREL 2021, 2021.
  78. J. Giner, R. Lamprecht, V. Gallina, C. Laflamme, L. Sielaff, and W. Sihn, “Demonstrating reinforcement learning for maintenance scheduling in a production environment,” in 2021 26th IEEE International Conference on Emerging Technologies and Factory Automation (ETFA), pp. 1–8, IEEE, 2021.
  79. P. D. Paraschos, G. K. Koulinas, and D. E. Koulouriotis, “Parametric and reinforcement learning control for degrading multi-stage systems,” Procedia Manufacturing, vol. 55, pp. 401–408, 2021.
  80. J. M. van Noortwijk, “A survey of the application of gamma processes in maintenance,” Reliability Engineering & System Safety, vol. 94, no. 1, pp. 2–21, 2009.
  81. A. J. Hallinan Jr, “A review of the weibull distribution,” Journal of Quality Technology, vol. 25, no. 2, pp. 85–93, 1993.
  82. P. P. Tambe and M. S. Kulkarni, “A reliability based integrated model of maintenance planning with quality control and production decision for improving operational performance,” Reliability Engineering & System Safety, vol. 226, p. 108681, 2022.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (2)
  1. Oluwaseyi Ogunfowora (4 papers)
  2. Homayoun Najjaran (40 papers)
Citations (32)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now