Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Natural Language Instructions for Intuitive Human Interaction with Robotic Assistants in Field Construction Work (2307.04195v2)

Published 9 Jul 2023 in cs.RO, cs.AI, and cs.HC

Abstract: The introduction of robots is widely considered to have significant potential of alleviating the issues of worker shortage and stagnant productivity that afflict the construction industry. However, it is challenging to use fully automated robots in complex and unstructured construction sites. Human-Robot Collaboration (HRC) has shown promise of combining human workers' flexibility and robot assistants' physical abilities to jointly address the uncertainties inherent in construction work. When introducing HRC in construction, it is critical to recognize the importance of teamwork and supervision in field construction and establish a natural and intuitive communication system for the human workers and robotic assistants. Natural language-based interaction can enable intuitive and familiar communication with robots for human workers who are non-experts in robot programming. However, limited research has been conducted on this topic in construction. This paper proposes a framework to allow human workers to interact with construction robots based on natural language instructions. The proposed method consists of three stages: Natural Language Understanding (NLU), Information Mapping (IM), and Robot Control (RC). Natural language instructions are input to a LLM to predict a tag for each word in the NLU module. The IM module uses the result of the NLU module and building component information to generate the final instructional output essential for a robot to acknowledge and perform the construction task. A case study for drywall installation is conducted to evaluate the proposed approach. The obtained results highlight the potential of using natural language-based interaction to replicate the communication that occurs between human workers within the context of human-robot teams.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (111)
  1. An agent-based approach for modeling human-robot collaboration in bricklaying. In ISARC. Proceedings of the International Symposium on Automation and Robotics in Construction, volume 37, pages 797–804. IAARC Publications, 2020.
  2. Earthmoving equipment automation: A review of technical advances and future outlook. Journal of Information Technology in Construction (ITcon), 22(13):247–265, 2017.
  3. Development of an automatic four-color spraying device carried by a robot arm. In Proceedings of 24th International Symposium on Automation and Robotics in construction ISARC, pages 19–21, 2007.
  4. Bio-inspired burrowing mechanism for underground locomotion control. In 30th International Symposium on Automation and Robotics in Construction and Mining, ISARC 2013, Held in Conjunction with the 23rd World Mining Congress, 2013.
  5. Andrzej Wieckowski. “ja-wa”-a wall construction system using unilateral material application with a mobile robot. Automation in Construction, 83:19–28, 2017.
  6. Tunnel structural inspection and assessment using an autonomous robotic system. Automation in Construction, 87:117–126, 2018.
  7. Cable-driven parallel robot for curtain wall modules automatic installation. In Proceedings of the 35th International Symposium on Automation and Robotics in Construction (ISARC), pages 396–403, 2018.
  8. Parallel 2-dof manipulator for wall-cleaning applications. Automation in Construction, 101:209–217, 2019.
  9. Robotics and automated systems in construction: Understanding industry-specific challenges for adoption. Journal of Building Engineering, 26:100868, 2019.
  10. Autonomous motion planning and task execution in geometrically adaptive robotized construction work. Automation in Construction, 100:24–45, 2019.
  11. Prediction-based path planning for safe and efficient human–robot collaboration in construction via deep reinforcement learning. Journal of Computing in Civil Engineering, 37(1):04022046, 2023.
  12. Vision guided autonomous robotic assembly and as-built scanning on unstructured construction sites. Automation in Construction, 59:128–138, 2015.
  13. Scene understanding for adaptive manipulation in robotized construction work. Automation in Construction, 82:16–30, 2017.
  14. Influencing factors of the future utilisation of construction robots for buildings: A hong kong perspective. Journal of Building Engineering, 30:101220, 2020.
  15. Teaching robots to perform quasi-repetitive construction tasks through human demonstration. Automation in Construction, 120:103370, 2020.
  16. Robo-partner: Seamless human-robot cooperation for intelligent, flexible and safe operations in the assembly factories of the future. Procedia CIRP, 23:71–76, 2014.
  17. Collaborative robots and industrial revolution 4.0 (ir 4.0). In 2020 International Conference on Emerging Trends in Smart Technologies (ICETST), pages 1–5. IEEE, 2020.
  18. Uncertainty-aware visualization and proximity monitoring in urban excavation: a geospatial augmented reality approach. Visualization in engineering, 1:1–13, 2013.
  19. A systematic approach to the engineering design of a hrc workcell for bio-medical product assembly. In 2015 IEEE 20th Conference on Emerging Technologies & Factory Automation (ETFA), pages 1–8. IEEE, 2015.
  20. Gerardo Cupido et al. The role of production and teamwork practices in construction safety: A cognitive model and an empirical case study. Journal of Safety Research, 40(4):265–275, 2009.
  21. Interactive and immersive process-level digital twin for collaborative human–robot construction work. Journal of Computing in Civil Engineering, 35(6):04021023, 2021.
  22. Survey on human–robot collaboration in industrial settings: Safety, intuitive interfaces and applications. Mechatronics, 55:248–266, 2018.
  23. Natural multimodal communication for human–robot collaboration. International Journal of Advanced Robotic Systems, 14(4):1729881417716043, 2017.
  24. New interaction metaphors to control a hydraulic working machine’s arm. In 2016 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR), pages 297–303. IEEE, 2016.
  25. Impact of vr-based training on human–robot interaction for remote operating construction robots. Journal of Computing in Civil Engineering, 36(3):04022006, 2022.
  26. Brain-computer interface for hands-free teleoperation of construction robots. Automation in Construction, 123:103523, 2021.
  27. Design and development of a novel robotic gripper for automated scaffolding assembly. In 2018 IEEE Third Ecuador Technical Chapters Meeting (ETCM), pages 1–6. IEEE, 2018.
  28. S Karpagavalli and Edy Chandra. A review on automatic speech recognition architecture and approaches. International Journal of Signal Processing, Image Processing and Pattern Recognition, 9(4):393–404, 2016.
  29. Human–robot interaction review and challenges on task planning and programming. International Journal of Computer Integrated Manufacturing, 29(8):916–931, 2016.
  30. Speech emotion recognition using 3d convolutions and attention-based sliding recurrent networks with auditory front-ends. IEEE Access, 8:16560–16572, 2020.
  31. A survey of robot learning strategies for human-robot collaboration in industrial settings. Robotics and Computer-Integrated Manufacturing, 73:102231, 2022.
  32. Very deep convolutional neural networks for noise robust speech recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 24(12):2263–2276, 2016.
  33. Optical laser microphone for human-robot interaction: speech recognition in extremely noisy service environments. Advanced Robotics, 36(5-6):304–317, 2022.
  34. Interactively picking real-world objects with unconstrained spoken language instructions. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 3774–3781. IEEE, 2018.
  35. A natural language instruction disambiguation method for robot grasping. In 2021 IEEE International Conference on Robotics and Biomimetics (ROBIO), pages 601–606. IEEE, 2021.
  36. Systems of natural-language-facilitated human-robot cooperation: A review. arXiv preprint arXiv:1701.08269, 2017.
  37. Natural-language-instructed industrial task execution. In International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, volume 50084, page V01BT02A043. American Society of Mechanical Engineers, 2016.
  38. Efficient grounding of abstract spatial concepts for natural language interaction with robot manipulators. 2016.
  39. Natural language communication with robots. pages 751–761, 2016.
  40. Understanding natural language instructions for fetching daily objects using gan-based multimodal target–source classification. IEEE Robotics and Automation Letters, 4(4):3884–3891, 2019.
  41. Bim-based simulation of construction robotics in the assembly process of wood frames. Automation in Construction, 137:104194, 2022.
  42. Human-drone interaction (hdi): Opportunities and considerations in construction. Automation and robotics in the architecture, engineering, and construction industry, pages 111–142, 2022.
  43. Vision–based framework for automatic interpretation of construction workers’ hand gestures. Automation in Construction, 130:103872, 2021.
  44. Hand gesture-based control of a front-end loader. In 2020 IEEE Canadian Conference on Electrical and Computer Engineering (CCECE), pages 1–4. IEEE, 2020.
  45. Foundations of visual linear human–robot interaction via pointing gesture navigation. International Journal of Social Robotics, 9:509–523, 2017.
  46. Scene graph and frame update algorithms for smooth and scalable 3d visualization of simulated construction operations. Computer-Aided Civil and Infrastructure Engineering, 17(4):228–245, 2002.
  47. Automated generation of dynamic, operations level virtual construction scenarios. Journal of Information Technology in Construction (ITcon), 8(6):65–84, 2003.
  48. Sensitivity analysis of augmented reality-assisted building damage reconnaissance using virtual prototyping. Automation in Construction, 33:24–36, 2013.
  49. A brief discussion on augmented reality and virtual reality in construction industry. Journal of System and Management Sciences, 7(3):1–33, 2017.
  50. Industrial robot control and operator training using virtual reality interfaces. Computers in Industry, 109:114–120, 2019.
  51. Intuitive robot teleoperation for civil engineering operations with virtual reality and deep learning scene reconstruction. Advanced Engineering Informatics, 46:101170, 2020.
  52. Integrated information modeling and visual simulation of engineering operations using dynamic augmented reality scene graphs. Journal of Information Technology in Construction (ITcon), 16(17):259–278, 2011.
  53. M Dalle Mura and G Dini. Augmented reality in assembly systems: state of the art and future perspectives. In Smart Technologies for Precision Assembly: 9th IFIP WG 5.5 International Precision Assembly Seminar, IPAS 2020, Virtual Event, December 14–15, 2020, Revised Selected Papers 9, pages 3–22. Springer, 2021.
  54. Review on existing vr/ar solutions in human–robot collaboration. Procedia CIRP, 97:407–411, 2021.
  55. A closed-loop brain-computer interface with augmented reality feedback for industrial human-robot collaboration. The International Journal of Advanced Manufacturing Technology, pages 1–16, 2021.
  56. Brainwave-driven human-robot collaboration in construction. Automation in Construction, 124:103556, 2021.
  57. Comprehensive review on brain-controlled mobile robots and robotic arms based on electroencephalography signals. Intelligent Service Robotics, 13:539–563, 2020.
  58. Artificial intelligence in the construction industry: A review of present status, opportunities and future challenges. Journal of Building Engineering, 44:103299, 2021.
  59. Guidelines for improving task-based natural language understanding in human-robot rescue teams. In 2017 8th IEEE International Conference on Cognitive Infocommunications (CogInfoCom), pages 000203–000208. IEEE, 2017.
  60. Learning object placements for relational instructions by hallucinating scene representations. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 94–100. IEEE, 2020.
  61. Target-dependent uniter: A transformer-based multimodal language comprehension model for domestic service robots. IEEE Robotics and Automation Letters, 6(4):8401–8408, 2021.
  62. Audio–visual language instruction understanding for robotic sorting. Robotics and Autonomous Systems, 159:104271, 2023.
  63. Following natural language instructions for household tasks with landmark guided search and reinforced pose adjustment. IEEE Robotics and Automation Letters, 7(3):6870–6877, 2022.
  64. Object-aware navigation for remote embodied visual referring expression. Neurocomputing, 515:68–78, 2023.
  65. Grounding robot plans from natural language instructions with incomplete world knowledge. In Conference on robot learning, pages 714–723. PMLR, 2018.
  66. Enabling robots to understand incomplete natural language instructions using commonsense reasoning. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 1963–1969. IEEE, 2020.
  67. Situated human–robot collaboration: predicting intent from grounded natural language. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 827–833. IEEE, 2018.
  68. Natural language processing: A review. International Journal of Research in Engineering and Applied Sciences, 6(3):207–210, 2016.
  69. Applications of natural language processing in construction. Automation in Construction, 136:104169, 2022.
  70. Retrieving similar cases for alternative dispute resolution in construction accidents using text mining techniques. Automation in construction, 34:85–91, 2013.
  71. Accident case retrieval and analyses: Using natural language processing in the construction industry. Journal of Construction Engineering and Management, 145(3):04019004, 2019.
  72. Automated content analysis for construction safety: A natural language processing system to extract precursors and outcomes from unstructured injury reports. Automation in Construction, 62:45–56, 2016.
  73. Extending building information models semiautomatically using semantic natural language processing techniques. Journal of Computing in Civil Engineering, 30(5):C4016004, 2016.
  74. An intelligent authoring model for subsidiary legislation and regulatory instrument drafting within construction and engineering industry. Automation in construction, 35:121–130, 2013.
  75. Effective risk positioning through automated identification of missing contract conditions from the contractor’s perspective based on fidic contract cases. Journal of Management in Engineering, 36(3):05020003, 2020.
  76. Natural language processing for smart construction: Current status and future directions. Automation in Construction, 134:104059, 2022.
  77. Use of keyphrase extraction software for creation of an aec/fm thesaurus. Journal of Information Technology in Construction (ITcon), 5(2):25–36, 2002.
  78. Holistic framework for highway construction cost index development based on inconsistent pay items. Journal of Construction Engineering and Management, 147(7):04021052, 2021.
  79. A multilabel classification approach to identify hurricane-induced infrastructure disruptions using social media data. Computer-Aided Civil and Infrastructure Engineering, 35(12):1387–1402, 2020.
  80. Deep learning-based extraction of construction procedural constraints from construction regulations. Advanced Engineering Informatics, 43:101003, 2020.
  81. Developing a hybrid approach to extract constraints related information for constraint management. Automation in Construction, 124:103563, 2021.
  82. Leveraging bim for automated fault detection in operational buildings. In ISARC. Proceedings of the International Symposium on Automation and Robotics in Construction, volume 30, page 1. Citeseer, 2013.
  83. Robotic data collection and simulation for evaluation of building retrofit performance. Automation in Construction, 92:88–102, 2018.
  84. Development of bim-integrated construction robot task planning and simulation system. Automation in Construction, 127:103720, 2021.
  85. A natural-language-based approach to intelligent data retrieval and representation for cloud bim. Computer-Aided Civil and Infrastructure Engineering, 31(1):18–33, 2016.
  86. Bimasr: framework for voice-based bim information retrieval. Journal of Construction Engineering and Management, 147(10):04021124, 2021.
  87. Nlp-based query-answering system for information extraction from building information models. Journal of Computing in Civil Engineering, 36(3):04022004, 2022.
  88. Unified transformer multi-task learning for intent classification with entity recognition. IEEE Access, 9:147306–147314, 2021.
  89. Framewise phoneme classification with bidirectional lstm and other neural network architectures. Neural networks, 18(5-6):602–610, 2005.
  90. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. 2001.
  91. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
  92. Bidirectional lstm-crf models for sequence tagging. arXiv preprint arXiv:1508.01991, 2015.
  93. Optimal hyperparameters for deep lstm-networks for sequence labeling tasks. arXiv preprint arXiv:1707.06799, 2017.
  94. Deep multi-task learning with cross connected layer for slot filling. In Natural Language Processing and Chinese Computing: 8th CCF International Conference, NLPCC 2019, Dunhuang, China, October 9–14, 2019, Proceedings, Part II 8, pages 308–317. Springer, 2019.
  95. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  96. Understanding the implications of digitisation and automation in the context of industry 4.0: A triangulation approach and elements of a research agenda for the construction industry. Computers in industry, 83:121–139, 2016.
  97. The 2021 annual construction technology report. 2021.
  98. Integration of infrastructure based positioning systems and inertial navigation for ubiquitous context-aware engineering applications. Advanced Engineering Informatics, 25(4):640–655, 2011.
  99. Moveit![ros topics]. IEEE Robotics & Automation Magazine, 19(1):18–19, 2012.
  100. HOME RenoVision DIY. How to install drywall a to z—diy tutorial. https://www.youtube.com/watch?v=VQIMaR7hWtM, 2020.
  101. Bim-based integrated approach for detailed construction scheduling under resource constraints. Automation in Construction, 53:29–43, 2015.
  102. Bim-based last planner system tool for improving construction project management. Automation in Construction, 104:246–254, 2019.
  103. An integrated bim-based approach for cost estimation in construction projects. Engineering, Construction and Architectural Management, 28(9):2828–2854, 2021.
  104. Reducing warehouse employee errors using voice-assisted technology that provided immediate feedback. Journal of Organizational Behavior Management, 27(1):1–31, 2007.
  105. Ergonomics improvement in a harsh environment using an audio feedback system. International Journal of Industrial Ergonomics, 40(6):767–774, 2010.
  106. David Goomas. Increasing warehouse worker performance using voice technology that provided immediate feedback: Personal performance productivity prompt. Journal of Organizational Behavior Management, pages 1–10, 2022.
  107. Recognizing chinese judicial named entity using bilstm-crf. In Journal of Physics: Conference Series, volume 1592, page 012040. IOP Publishing, 2020.
  108. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  109. Empirical study of deep learning for text classification in legal document review. In 2018 IEEE International Conference on Big Data (Big Data), pages 3317–3320. IEEE, 2018.
  110. Deep learning techniques: an overview. Advanced Machine Learning Technologies and Applications: Proceedings of AMLTA 2020, pages 599–608, 2021.
  111. Aysu Ezen-Can. A comparison of lstm and bert for small corpus. arXiv preprint arXiv:2009.05451, 2020.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Somin Park (3 papers)
  2. Xi Wang (275 papers)
  3. Carol C. Menassa (7 papers)
  4. Vineet R. Kamat (8 papers)
  5. Joyce Y. Chai (7 papers)
Citations (12)
View on Semantic Scholar