Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Hierarchical Deep Learning for Intention Estimation of Teleoperation Manipulation in Assembly Tasks (2403.19770v1)

Published 28 Mar 2024 in cs.RO, cs.AI, and cs.LG

Abstract: In human-robot collaboration, shared control presents an opportunity to teleoperate robotic manipulation to improve the efficiency of manufacturing and assembly processes. Robots are expected to assist in executing the user's intentions. To this end, robust and prompt intention estimation is needed, relying on behavioral observations. The framework presents an intention estimation technique at hierarchical levels i.e., low-level actions and high-level tasks, by incorporating multi-scale hierarchical information in neural networks. Technically, we employ hierarchical dependency loss to boost overall accuracy. Furthermore, we propose a multi-window method that assigns proper hierarchical prediction windows of input data. An analysis of the predictive power with various inputs demonstrates the predominance of the deep hierarchical model in the sense of prediction accuracy and early intention identification. We implement the algorithm on a virtual reality (VR) setup to teleoperate robotic hands in a simulation with various assembly tasks to show the effectiveness of online estimation.

View on arXiv
Definition Search Book Streamline Icon: https://streamlinehq.com
References (39)
  1. K. I. Alevizos, C. P. Bechlioulis, and K. J. Kyriakopoulos, “Physical human–robot cooperation based on robust motion intention estimation,” Robotica, vol. 38, no. 10, pp. 1842–1866, 2020.
  2. C. Fang, L. Peternel, A. Seth, M. Sartori, K. Mombaur, and E. Yoshida, “Human modeling in physical human-robot interaction: A brief survey,” IEEE Robotics and Automation Letters, 2023.
  3. R. Wilcox, S. Nikolaidis, and J. Shah, “Optimization of temporal dynamics for adaptive human-robot interaction in assembly manufacturing,” Robotics, vol. 8, no. 441, pp. 10–15, 2013.
  4. Z. Zhou, S. Wang, Z. Chen, M. Cai, H. Wang, Z. Li, and Z. Kan, “Local observation based reactive temporal logic planning of human-robot systems,” IEEE Transactions on Automation Science and Engineering, 2023.
  5. G. Li, Z. Li, and Z. Kan, “Assimilation control of a robotic exoskeleton for physical human-robot interaction,” IEEE Robotics and Automation Letters, vol. 7, no. 2, pp. 2977–2984, 2022.
  6. P. Schydlo, M. Rakovic, L. Jamone, and J. Santos-Victor, “Anticipation in human-robot cooperation: A recurrent neural network approach for multiple action sequences prediction,” in 2018 IEEE International Conference on Robotics and Automation (ICRA).   IEEE, 2018, pp. 5909–5914.
  7. D. Nicolis, A. M. Zanchettin, and P. Rocco, “Human intention estimation based on neural networks for enhanced collaboration with robots,” in 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).   IEEE, 2018, pp. 1326–1333.
  8. A. Belardinelli, A. R. Kondapally, D. Ruiken, D. Tanneberg, and T. Watabe, “Intention estimation from gaze and motion features for human-robot shared-control object manipulation,” in 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).   IEEE, 2022, pp. 9806–9813.
  9. Z. Huang, Y.-J. Mun, X. Li, Y. Xie, N. Zhong, W. Liang, J. Geng, T. Chen, and K. Driggs-Campbell, “Hierarchical intention tracking for robust human-robot collaboration in industrial assembly tasks,” in 2023 IEEE International Conference on Robotics and Automation (ICRA).   IEEE, 2023, pp. 9821–9828.
  10. S. Manschitz and D. Ruiken, “Shared autonomy for intuitive teleoperation,” ICRA Workshop: Shared Autonomy in Physical Human-Robot Interaction: Adaptability and Trust, May 2022.
  11. D. Gao, W. Yang, H. Zhou, Y. Wei, Y. Hu, and H. Wang, “Deep hierarchical classification for category prediction in e-commerce system,” arXiv preprint arXiv:2005.06692, 2020.
  12. T. B. Sheridan, “Teleoperation, telerobotics and telepresence: A progress report,” Control Engineering Practice, vol. 3, no. 2, pp. 205–214, 1995.
  13. L. Rozo, S. Calinon, D. G. Caldwell, P. Jimenez, and C. Torras, “Learning physical collaborative robot behaviors from human demonstrations,” IEEE Transactions on Robotics, vol. 32, no. 3, pp. 513–527, 2016.
  14. W. Yu, R. Alqasemi, R. Dubey, and N. Pernalete, “Telemanipulation assistance based on motion intention recognition,” in Proceedings of the 2005 IEEE international conference on robotics and automation.   IEEE, 2005, pp. 1121–1126.
  15. A. D. Dragan and S. S. Srinivasa, “A policy-blending formalism for shared control,” The International Journal of Robotics Research, vol. 32, no. 7, pp. 790–805, 2013.
  16. K. Hauser, “Recognition, prediction, and planning for assisted teleoperation of freeform tasks,” Autonomous Robots, vol. 35, pp. 241–254, 2013.
  17. D. Aarno and D. Kragic, “Motion intention recognition in robot assisted applications,” Robotics and Autonomous Systems, vol. 56, no. 8, pp. 692–705, 2008.
  18. A. K. Tanwani and S. Calinon, “A generative model for intention recognition and manipulation assistance in teleoperation,” in 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).   IEEE, 2017, pp. 43–50.
  19. L. R. Rabiner, “A tutorial on hidden markov models and selected applications in speech recognition,” Proceedings of the IEEE, vol. 77, no. 2, pp. 257–286, 1989.
  20. A. K. Tanwani and S. Calinon, “Learning robot manipulation tasks with task-parameterized semitied hidden semi-markov model,” IEEE Robotics and Automation Letters, vol. 1, no. 1, pp. 235–242, 2016.
  21. C. Zhu, Q. Cheng, and W. Sheng, “Human intention recognition in smart assisted living systems using a hierarchical hidden markov model,” in 2008 IEEE International Conference on Automation Science and Engineering.   IEEE, 2008, pp. 253–258.
  22. Y. Li and S. S. Ge, “Human–robot collaboration based on motion intention estimation,” IEEE/ASME Transactions on Mechatronics, vol. 19, no. 3, pp. 1007–1014, 2013.
  23. E. De Momi, L. Kranendonk, M. Valenti, N. Enayati, and G. Ferrigno, “A neural network-based approach for trajectory planning in robot–human handover tasks,” Frontiers in Robotics and AI, vol. 3, p. 34, 2016.
  24. C. Lea, R. Vidal, and G. D. Hager, “Learning convolutional action primitives for fine-grained action recognition,” in 2016 IEEE international conference on robotics and automation (ICRA).   IEEE, 2016, pp. 1642–1649.
  25. C. Yuan, T. Marion, and M. Moghaddam, “Leveraging end-user data for enhanced design concept evaluation: A multimodal deep regression model,” Journal of Mechanical Design, vol. 144, no. 2, p. 021403, 2022.
  26. W. Lu, Z. Hu, and J. Pan, “Human-robot collaboration using variable admittance control and human intention prediction,” in 2020 IEEE 16th International Conference on Automation Science and Engineering (CASE).   IEEE, 2020, pp. 1116–1121.
  27. J. Bandouch and M. Beetz, “Tracking humans interacting with the environment using efficient hierarchical sampling and layered observation models,” in 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops.   IEEE, 2009, pp. 2040–2047.
  28. B. Hayes and B. Scassellati, “Autonomously constructing hierarchical task networks for planning and human-robot collaboration,” in 2016 IEEE International Conference on Robotics and Automation (ICRA).   IEEE, 2016, pp. 5469–5476.
  29. S. Holtzen, Y. Zhao, T. Gao, J. B. Tenenbaum, and S.-C. Zhu, “Inferring human intent from video by sampling hierarchical plans,” in 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).   IEEE, 2016, pp. 1489–1496.
  30. J.-H. Han, S.-H. Choi, and J.-H. Kim, “Interactive human intention reading by learning hierarchical behavior knowledge networks for human-robot interaction,” ETRI Journal, vol. 38, no. 6, pp. 1229–1239, 2016.
  31. C. Feichtenhofer, H. Fan, J. Malik, and K. He, “Slowfast networks for video recognition,” in Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 6202–6211.
  32. L. R. Medsker and L. Jain, “Recurrent neural networks,” Design and Applications, vol. 5, no. 64-67, p. 2, 2001.
  33. A. Graves and A. Graves, “Long short-term memory,” Supervised sequence labelling with recurrent neural networks, pp. 37–45, 2012.
  34. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems, vol. 30, 2017.
  35. F. Abi-Farraj, T. Osa, N. P. J. Peters, G. Neumann, and P. R. Giordano, “A learning-based shared control architecture for interactive task execution,” in 2017 IEEE international conference on robotics and automation (ICRA).   IEEE, 2017, pp. 329–335.
  36. J. T. Feddema and O. R. Mitchell, “Vision-guided servoing with feature-based trajectory generation (for robots),” IEEE Transactions on Robotics and Automation, vol. 5, no. 5, pp. 691–700, 1989.
  37. T. Marcucci, M. Petersen, D. von Wrangel, and R. Tedrake, “Motion planning around obstacles with convex optimization,” arXiv preprint arXiv:2205.04422, 2022.
  38. L. X. Shi, A. Sharma, T. Z. Zhao, and C. Finn, “Waypoint-based imitation learning for robotic manipulation,” arXiv preprint arXiv:2307.14326, 2023.
  39. L. Rabiner and B. Juang, “An introduction to hidden markov models,” ieee assp magazine, vol. 3, no. 1, pp. 4–16, 1986.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Mingyu Cai (21 papers)
  2. Karankumar Patel (3 papers)
  3. Soshi Iba (15 papers)
  4. Songpo Li (4 papers)
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets