Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
125 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Digital Twin-Driven Reinforcement Learning for Obstacle Avoidance in Robot Manipulators: A Self-Improving Online Training Framework (2403.13090v1)

Published 19 Mar 2024 in cs.RO, cs.SY, and eess.SY

Abstract: The evolution and growing automation of collaborative robots introduce more complexity and unpredictability to systems, highlighting the crucial need for robot's adaptability and flexibility to address the increasing complexities of their environment. In typical industrial production scenarios, robots are often required to be re-programmed when facing a more demanding task or even a few changes in workspace conditions. To increase productivity, efficiency and reduce human effort in the design process, this paper explores the potential of using digital twin combined with Reinforcement Learning (RL) to enable robots to generate self-improving collision-free trajectories in real time. The digital twin, acting as a virtual counterpart of the physical system, serves as a 'forward run' for monitoring, controlling, and optimizing the physical system in a safe and cost-effective manner. The physical system sends data to synchronize the digital system through the video feeds from cameras, which allows the virtual robot to update its observation and policy based on real scenarios. The bidirectional communication between digital and physical systems provides a promising platform for hardware-in-the-loop RL training through trial and error until the robot successfully adapts to its new environment. The proposed online training framework is demonstrated on the Unfactory Xarm5 collaborative robot, where the robot end-effector aims to reach the target position while avoiding obstacles. The experiment suggest that proposed framework is capable of performing policy online training, and that there remains significant room for improvement.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (36)
  1. A. A. Malik and A. Brem, “Man, machine and work in a digital twin setup: a case study,” ArXiv, vol. abs/2006.08760, 2020. [Online]. Available: https://api.semanticscholar.org/CorpusID:219792071
  2. I. Elguea-Aguinaco, A. Serrano-Munoz, D. Chrysostomou, I. Inziarte-Hidalgo, S. Bogh, and N. Arana-Arexolaleiba, “Goal-conditioned reinforcement learning within a human-robot disassembly environment,” Applied Sciences, vol. 12, no. 22, 2022.
  3. D. Kim, M. Choi, and J. Um, “Digital twin for autonomous collaborative robot by using synthetic data and reinforcement learning,” Robotics and Computer-Integrated Manufacturing, vol. 85, 2024.
  4. L. Ren, J. Dong, D. Huang, and J. Lü, “Digital twin robotic system with continuous learning for grasp detection in variable scenes,” IEEE Transactions on Industrial Electronics, pp. 1–11, 2024.
  5. S. Wang, J. Zhang, P. Wang, J. Law, R. Calinescu, and L. Mihaylova, “A deep learning-enhanced digital twin framework for improving safety and reliability in human–robot collaborative manufacturing,” Robotics and Computer-Integrated Manufacturing, vol. 85, 2024.
  6. Z. Zhang, Z. Zhang, L. Wang, X. Zhu, H. Huang, and Q. Cao, “Digital twin-enabled grasp outcomes assessment for unknown objects using visual-tactile fusion perception,” Robotics and Computer-Integrated Manufacturing, vol. 84, 2023.
  7. J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel, “Domain randomization for transferring deep neural networks from simulation to the real world,” in 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2017, pp. 23–30.
  8. D. Liu, Y. Chen, and Z. Wu, “Digital twin (dt)-cyclegan: Enabling zero-shot sim-to-real transfer of visual grasping models,” IEEE Robotics and Automation Letters, vol. 8, no. 5, pp. 2421–2428, 2023.
  9. M. Müller, T. Ruppert, N. Jazdi, and M. Weyrich, “Self-improving situation awareness for human–robot-collaboration using intelligent digital twin,” Journal of Intelligent Manufacturing, 2023.
  10. E. Coumans and Y. Bai, “Pybullet, a python module for physics simulation for games, robotics and machine learning,” http://pybullet.org, 2016–2021.
  11. F. P. Audonnet, A. Hamilton, and G. Aragon-Camarasa, “A systematic comparison of simulation software for robotic arm manipulation using ros2,” in 2022 22nd International Conference on Control, Automation and Systems (ICCAS).   IEEE, 2022, pp. 755–762.
  12. D. Mukherjee, K. Gupta, L. H. Chang, and H. Najjaran, “A survey of robot learning strategies for human-robot collaboration in industrial settings,” Robotics and Computer-Integrated Manufacturing, vol. 73, 2022.
  13. Z. Liu, Q. Wang, B. Yang, and K. Rajakani, “Reinforcement learning-based path planning algorithm for mobile robots,” Wireless Communications and Mobile Computing, vol. 2022, pp. 1–10, 2022.
  14. J. Luo, E. Solowjow, C. Wen, J. A. Ojea, A. M. Agogino, A. Tamar, and P. Abbeel, “Reinforcement learning on variable impedance controller for high-precision robotic assembly,” in 2019 International Conference on Robotics and Automation (ICRA).   IEEE, Conference Proceedings, pp. 3080–3087.
  15. B. Sangiovanni, A. Rendiniello, G. P. Incremona, A. Ferrara, and M. Piastra, “Deep reinforcement learning for collision avoidance of robotic manipulators,” in 2018 European Control Conference (ECC), Conference Proceedings, pp. 2063–2068.
  16. T. Yu, J. Huang, and Q. Chang, “Optimizing task scheduling in human-robot collaboration with deep multi-agent reinforcement learning,” Journal of Manufacturing Systems, vol. 60, pp. 487–499, 2021.
  17. C. Li, P. Zheng, Y. Yin, Y. M. Pang, and S. Huo, “An ar-assisted deep reinforcement learning-based approach towards mutual-cognitive safe human-robot interaction,” Robotics and Computer-Integrated Manufacturing, vol. 80, 2023.
  18. Q. Liu, Z. Liu, B. Xiong, W. Xu, and Y. Liu, “Deep reinforcement learning-based safe interaction for industrial human-robot collaboration using intrinsic reward function,” Advanced Engineering Informatics, vol. 49, 2021.
  19. J. Luo, O. Sushkov, R. Pevceviciute, W. Lian, C. Su, M. Vecerik, N. Ye, S. Schaal, and J. Scholz, “Robust multi-modal policies for industrial assembly via reinforcement learning and demonstrations: A large-scale study,” arXiv preprint arXiv:2103.11512, 2021.
  20. X. Zhao, T. Fan, Y. Li, Y. Zheng, and J. Pan, “An efficient and responsive robot motion controller for safe human-robot collaboration,” IEEE Robotics and Automation Letters, vol. 6, no. 3, pp. 6068–6075, 2021.
  21. D. Jones, C. Snider, A. Nassehi, J. Yon, and B. Hicks, “Characterising the digital twin: A systematic literature review,” CIRP Journal of Manufacturing Science and Technology, vol. 29, pp. 36–52, 2020.
  22. M. Grieves, “Digital twin: manufacturing excellence through virtual factory replication,” White paper, vol. 1, no. 2014, pp. 1–7, 2014.
  23. M. Attaran, S. Attaran, and B. G. Celik, “The impact of digital twins on the evolution of intelligent manufacturing and industry 4.0,” Adv Comput Intell, vol. 3, no. 3, p. 11, 2023. [Online]. Available: https://www.ncbi.nlm.nih.gov/pubmed/37305021
  24. W. Hu, C. Wang, F. Liu, X. Peng, P. Sun, and J. Tan, “A grasps-generation-and-selection convolutional neural network for a digital twin of intelligent robotic grasping,” Robotics and Computer-Integrated Manufacturing, vol. 77, 2022.
  25. Y. Liu, H. Xu, D. Liu, and L. Wang, “A digital twin-based sim-to-real transfer for deep reinforcement learning-enabled industrial robot grasping,” Robotics and Computer-Integrated Manufacturing, vol. 78, 2022.
  26. T. Delbrügger, L. T. Lenz, D. Losch, and J. Roßmann, “A navigation framework for digital twins of factories based on building information modeling,” in 2017 22nd IEEE International Conference on Emerging Technologies and Factory Automation (ETFA), 2017, pp. 1–4.
  27. Y. Du, Y. Luo, Y. Peng, and Y. Chen, “Industrial robot digital twin system motion simulation and collision detection,” pp. 1–4, 2021.
  28. J. Liu, Z. Xu, H. Xiong, Q. Lin, W. Xu, and Z. Zhou, “Digital twin-driven robotic disassembly sequence dynamic planning under uncertain missing condition,” IEEE Transactions on Industrial Informatics, pp. 1–9, 2023.
  29. D. Lee, S. Lee, N. Masoud, M. S. Krishnan, and V. C. Li, “Digital twin-driven deep reinforcement learning for adaptive task allocation in robotic construction,” Advanced Engineering Informatics, vol. 53, 2022.
  30. J. Li, B. Shang, I. Jayawardana, and G. Chen, “Hardware-in-the-loop and digital twin enabled autonomous robotics-assisted environment inspection*,” pp. 1–5, 2023.
  31. R. S. S. Barto and A. G., “Reinforcement learning an introduction,” 2014.
  32. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” arXiv preprint arXiv:1707.06347, 2017.
  33. A. Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Ernestus, and N. Dormann, “Stable-baselines3: Reliable reinforcement learning implementations,” Journal of Machine Learning Research, vol. 22, no. 268, pp. 1–8, 2021. [Online]. Available: http://jmlr.org/papers/v22/20-1364.html
  34. G. Jocher, A. Chaurasia, and J. Qiu, “Ultralytics YOLO,” Jan. 2023. [Online]. Available: https://github.com/ultralytics/ultralytics
  35. “A object detection datasets for xarm5 robot in roboflow,” https://universe.roboflow.com/xarmgym/xarm_cubes, accessed: 2023-11-29.
  36. Y. Sun, “Digital twin-driven reinforcement learning for obstacle avoidance in robot manipulators: A self-improving online training framework,” https://youtu.be/CMgg1rwiUTo, 2016–2021.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now