Integrating Uncertainty-Aware Human Motion Prediction into Graph-Based Manipulator Motion Planning (2405.09779v1)
Abstract: There has been a growing utilization of industrial robots as complementary collaborators for human workers in re-manufacturing sites. Such a human-robot collaboration (HRC) aims to assist human workers in improving the flexibility and efficiency of labor-intensive tasks. In this paper, we propose a human-aware motion planning framework for HRC to effectively compute collision-free motions for manipulators when conducting collaborative tasks with humans. We employ a neural human motion prediction model to enable proactive planning for manipulators. Particularly, rather than blindly trusting and utilizing predicted human trajectories in the manipulator planning, we quantify uncertainties of the neural prediction model to further ensure human safety. Moreover, we integrate the uncertainty-aware prediction into a graph that captures key workspace elements and illustrates their interconnections. Then a graph neural network is leveraged to operate on the constructed graph. Consequently, robot motion planning considers both the dependencies among all the elements in the workspace and the potential influence of future movements of human workers. We experimentally validate the proposed planning framework using a 6-degree-of-freedom manipulator in a shared workspace where a human is performing disassembling tasks. The results demonstrate the benefits of our approach in terms of improving the smoothness and safety of HRC. A brief video introduction of this work is available as the supplemental materials.
- M.-L. Lee, W. Liu, S. Behdad, X. Liang, and M. Zheng, “Robot-assisted disassembly sequence planning with real-time human motion prediction,” IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 53, no. 1, pp. 438–450, 2022.
- M.-L. Lee, X. Liang, B. Hu, G. Onel, S. Behdad, and M. Zheng, “A review of prospects and opportunities in disassembly with human–robot collaboration,” Journal of Manufacturing Science and Engineering, vol. 146, no. 2, 2024.
- H. Liu and L. Wang, “Human motion prediction for human-robot collaboration,” Journal of Manufacturing Systems, vol. 44, pp. 287–294, 2017.
- D. Fridovich-Keil, A. Bajcsy, J. F. Fisac, S. L. Herbert, S. Wang, A. D. Dragan, and C. J. Tomlin, “Confidence-aware motion prediction for real-time collision avoidance1,” The International Journal of Robotics Research, vol. 39, no. 2-3, pp. 250–265, 2020.
- S. Tian, M. Zheng, and X. Liang, “Transfusion: A practical and effective transformer-based diffusion model for 3d human motion prediction,” IEEE Robotics and Automation Letters, Accepted, 2024.
- K. A. Eltouny, W. Liu, S. Tian, M. Zheng, and X. Liang, “De-tgn: Uncertainty-aware human motion forecasting using deep ensembles,” IEEE Robotics and Automation Letters, 2024.
- V. V. Unhelkar, P. A. Lasota, Q. Tyroller, R.-D. Buhai, L. Marceau, B. Deml, and J. A. Shah, “Human-aware robotic assistant for collaborative assembly: Integrating human motion prediction with planning in time,” IEEE Robotics and Automation Letters, vol. 3, no. 3, pp. 2394–2401, 2018.
- H. Moudoud, Z. Mlika, L. Khoukhi, and S. Cherkaoui, “Detection and prediction of fdi attacks in iot systems via hidden markov model,” IEEE Transactions on Network Science and Engineering, vol. 9, no. 5, pp. 2978–2990, 2022.
- J. S. Park, C. Park, and D. Manocha, “I-planner: Intention-aware motion planning using learning-based human motion prediction,” The International Journal of Robotics Research, vol. 38, no. 1, pp. 23–39, 2019.
- W. Liu, X. Liang, and M. Zheng, “Dynamic model informed human motion prediction based on unscented kalman filter,” IEEE/ASME Transactions on Mechatronics, vol. 27, no. 6, pp. 5287–5295, 2022.
- M. Li, S. Chen, Y. Zhao, Y. Zhang, Y. Wang, and Q. Tian, “Dynamic multiscale graph neural networks for 3d skeleton based human motion prediction,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 214–223.
- A. Mohamed, K. Qian, M. Elhoseiny, and C. Claudel, “Social-stgcnn: A social spatio-temporal graph convolutional neural network for human trajectory prediction,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 14 424–14 432.
- E. Aksan, M. Kaufmann, P. Cao, and O. Hilliges, “A spatio-temporal transformer for 3d human motion prediction,” in 2021 International Conference on 3D Vision (3DV). IEEE, 2021, pp. 565–574.
- J. Wang, H. Xu, M. Narasimhan, and X. Wang, “Multi-person 3d motion prediction with multi-range transformers,” Advances in Neural Information Processing Systems, vol. 34, pp. 6036–6049, 2021.
- M. Faroni, M. Beschi, and N. Pedrocchi, “Safety-aware time-optimal motion planning with uncertain human state estimation,” IEEE Robotics and Automation Letters, vol. 7, no. 4, pp. 12 219–12 226, 2022.
- A. Kanazawa, J. Kinugawa, and K. Kosuge, “Adaptive motion planning for a collaborative robot based on prediction uncertainty to enhance human safety and work efficiency,” IEEE Transactions on Robotics, vol. 35, no. 4, pp. 817–832, 2019.
- Y. Cheng, W. Zhao, C. Liu, and M. Tomizuka, “Human motion prediction using semi-adaptable neural networks,” in 2019 American Control Conference (ACC). IEEE, 2019, pp. 4884–4890.
- J. Zhang, D.-P. Fan, Y. Dai, S. Anwar, F. S. Saleh, T. Zhang, and N. Barnes, “Uc-net: Uncertainty inspired rgb-d saliency detection via conditional variational autoencoders,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 8582–8591.
- P. E. Hart, N. J. Nilsson, and B. Raphael, “A formal basis for the heuristic determination of minimum cost paths,” IEEE transactions on Systems Science and Cybernetics, vol. 4, no. 2, pp. 100–107, 1968.
- S. M. LaValle et al., “Rapidly-exploring random trees: A new tool for path planning,” 1998.
- J. D. Gammell, S. S. Srinivasa, and T. D. Barfoot, “Batch informed trees (bit*): Sampling-based optimal planning via the heuristically guided search of implicit random geometric graphs,” in 2015 IEEE international conference on robotics and automation (ICRA). IEEE, 2015, pp. 3067–3074.
- L. Janson, E. Schmerling, A. Clark, and M. Pavone, “Fast marching tree: A fast marching sampling-based method for optimal motion planning in many dimensions,” The International journal of robotics research, vol. 34, no. 7, pp. 883–921, 2015.
- T. Marcucci, M. Petersen, D. von Wrangel, and R. Tedrake, “Motion planning around obstacles with convex optimization,” Science Robotics, vol. 8, no. 84, p. eadf7843, 2023.
- S. Zimmermann, G. Hakimifard, M. Zamora, R. Poranne, and S. Coros, “A multi-level optimization framework for simultaneous grasping and motion planning,” IEEE Robotics and Automation Letters, vol. 5, no. 2, pp. 2966–2972, 2020.
- L. Li, Y. Miao, A. H. Qureshi, and M. C. Yip, “Mpc-mpnet: Model-predictive motion planning networks for fast, near-optimal planning under kinodynamic constraints,” IEEE Robotics and Automation Letters, vol. 6, no. 3, pp. 4496–4503, 2021.
- M. J. Bency, A. H. Qureshi, and M. C. Yip, “Neural path planning: Fixed time, near-optimal path generation via oracle imitation,” in 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2019, pp. 3965–3972.
- A. H. Qureshi, Y. Miao, A. Simeonov, and M. C. Yip, “Motion planning networks: Bridging the gap between learning-based and classical motion planners,” IEEE Transactions on Robotics, vol. 37, no. 1, pp. 48–66, 2020.
- P. Cai, H. Wang, H. Huang, Y. Liu, and M. Liu, “Vision-based autonomous car racing using deep imitative reinforcement learning,” IEEE Robotics and Automation Letters, vol. 6, no. 4, pp. 7262–7269, 2021.
- L. Gao, Z. Gu, C. Qiu, L. Lei, S. E. Li, S. Zheng, W. Jing, and J. Chen, “Cola-hrl: Continuous-lattice hierarchical reinforcement learning for autonomous driving,” in 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2022, pp. 13 143–13 150.
- W. Liu, K. Eltouny, S. Tian, X. Liang, and M. Zheng, “Kg-planner: Knowledge-informed graph neural planning for collaborative manipulators,” arXiv preprint arXiv:2405.07962, 2024.
- Y. Cheng, L. Sun, C. Liu, and M. Tomizuka, “Towards efficient human-robot collaboration with robust plan recognition and trajectory prediction,” IEEE Robotics and Automation Letters, vol. 5, no. 2, pp. 2602–2609, 2020.
- P. Kratzer, M. Toussaint, and J. Mainprice, “Prediction of human full-body movements with motion optimization and recurrent neural networks,” in 2020 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2020, pp. 1792–1798.
- P. Zheng, P.-B. Wieber, J. Baber, and O. Aycard, “Human arm motion prediction for collision avoidance in a shared workspace,” Sensors, vol. 22, no. 18, p. 6951, 2022.
- Y. Gal and Z. Ghahramani, “Dropout as a bayesian approximation: Representing model uncertainty in deep learning,” in international conference on machine learning. PMLR, 2016, pp. 1050–1059.
- Y. Gal and Z. Ghahramani, “Bayesian convolutional neural networks with bernoulli approximate variational inference,” arXiv preprint arXiv:1506.02158, 2015.
- S. Karaman and E. Frazzoli, “Sampling-based algorithms for optimal motion planning,” The international journal of robotics research, vol. 30, no. 7, pp. 846–894, 2011.
- J. A. Starek, J. V. Gomez, E. Schmerling, L. Janson, L. Moreno, and M. Pavone, “An asymptotically-optimal sampling-based algorithm for bi-directional motion planning,” in 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2015, pp. 2072–2078.
- Wansong Liu (7 papers)
- Kareem Eltouny (5 papers)
- Sibo Tian (14 papers)
- Xiao Liang (132 papers)
- Minghui Zheng (20 papers)