Papers
Topics
Authors
Recent
Search
2000 character limit reached

SOCRATES: Text-based Human Search and Approach using a Robot Dog

Published 10 Feb 2023 in cs.RO and cs.AI | (2302.05324v2)

Abstract: In this paper, we propose a SOCratic model for Robots Approaching humans based on TExt System (SOCRATES) focusing on the human search and approach based on free-form textual description; the robot first searches for the target user, then the robot proceeds to approach in a human-friendly manner. In particular, textual descriptions are composed of appearance (e.g., wearing white shirts with black hair) and location clues (e.g., is a student who works with robots). We initially present a Human Search Socratic Model that connects large pre-trained models in the language domain to solve the downstream task, which is searching for the target person based on textual descriptions. Then, we propose a hybrid learning-based framework for generating target-cordial robotic motion to approach a person, consisting of a learning-from-demonstration module and a knowledge distillation module. We validate the proposed searching module via simulation using a virtual mobile robot as well as through real-world experiments involving participants and the Boston Dynamics Spot robot. Furthermore, we analyze the properties of the proposed approaching framework with human participants based on the Robotic Social Attributes Scale (RoSAS)

Definition Search Book Streamline Icon: https://streamlinehq.com
References (37)
  1. The robotic social attributes scale (rosas): Development and validation. In Proc. of the 2017 12th ACM/IEEE International Conference on Human-Robot Interaction (HRI, pages 254–262, 2017.
  2. Language models are few-shot learners. In Proc. of the Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 1877–1901, 2020.
  3. Learning transferable visual models from natural language supervision. In Proc. of the International Conference on Machine Learning (ICML), pages 8748–8763. PMLR, 2021.
  4. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022.
  5. Bc-z: Zero-shot task generalization with robotic imitation learning. In Proc. of the Conference on Robot Learning (CoRL), pages 991–1002. PMLR, 2022.
  6. Vision-language navigation with self-supervised auxiliary reasoning tasks. In Proc. of the Conference on Computer Vision and Pattern Recognition (CVPR), pages 10012–10022, 2020.
  7. Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action. In Proc. of the Conference on Robot Learning (CoRL), 2022.
  8. Clip on wheels: Zero-shot object navigation as object localization and exploration. arXiv preprint arXiv:2203.10421, 2022.
  9. Simple but effective: Clip embeddings for embodied ai. In Proc. of the Conference on Computer Vision and Pattern Recognition (CVPR), pages 14829–14838, 2022.
  10. Zson: Zero-shot object-goal navigation using multimodal goal embeddings. arXiv preprint arXiv:2206.12403, 2022.
  11. Zero-shot active visual search (zavis): Intelligent object search for robotic assistants. In Proc. of the IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023.
  12. Socratic models: Composing zero-shot multimodal reasoning with language. In Proc. of the International Conference on Learning Representations (ICLR), 2022.
  13. Inner monologue: Embodied reasoning through planning with language models. In Proc. of the Conference on Robot Learning (CoRL), 2022.
  14. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proc. of the IEEE international conference on computer vision (ICCV), pages 618–626, 2017.
  15. Brian Yamauchi. Frontier-based exploration using multiple robots. In Proc. of the International conference on Autonomous agents, pages 47–53, 1998.
  16. Online learning to approach a person with no regret. IEEE Robotics and Automation Letters, 3(1):52–59, 2017.
  17. Learning goal conditioned socially compliant navigation from demonstration using risk-based features. IEEE Robotics and Automation Letters, 6(2):651–658, 2021.
  18. Robot navigation in crowds by graph convolutional networks with attention learned from human gaze. IEEE Robotics and Automation Letters, 5(2):2754–2761, 2020.
  19. Plug-and-play vqa: Zero-shot vqa by conjoining large pretrained models with zero training. In Proc. of the Empirical Methods in Natural Language Processing (EMNLP), 2022.
  20. Efficient estimation of word representations in vector space. In Proc. of the International Conference on Learning Representations (ICLR), 2013.
  21. Glenn Jocher. ultralytics/yolov5: v3.1 - Bug Fixes and Performance Improvements. https://github.com/ultralytics/yolov5, October 2020.
  22. Microsoft coco: Common objects in context. In Proc. of the European conference on computer vision (ECCV), pages 740–755. Springer, 2014.
  23. Mediapipe: A framework for building perception pipelines. arXiv preprint arXiv:1906.08172, 2019.
  24. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In Proc. of the International Conference on Machine Learning (ICML)), 2022.
  25. Density matching reward learning. arXiv preprint arXiv:1608.03694, 2016.
  26. Fast marching tree: A fast marching sampling-based method for optimal motion planning in many dimensions. The International journal of robotics research, 34(7):883–921, 2015.
  27. The Open Motion Planning Library. IEEE Robotics & Automation Magazine, 19(4):72–82, December 2012. https://ompl.kavrakilab.org.
  28. Efficient dynamic object search in home environment by mobile robot: A priori knowledge-based approach. IEEE Transactions on Vehicular Technology, 68(10):9466–9477, 2019.
  29. OpenCV Luxonis (n.d.). Opencv ai kit, July 2021.
  30. On evaluation of embodied navigation agents. arXiv preprint arXiv:1807.06757, 2018.
  31. Utilizing object-object and object-scene context when planning to find things. In 2009 IEEE International Conference on Robotics and Automation, pages 2168–2173, 2009.
  32. Simple open-vocabulary object detection with vision transformers. In Proc. of the European Conference on Computer Vision (ECCV), 2022.
  33. Density estimation through convex combinations of densities: approximation and estimation bounds. Neural Networks, 10(1):99–109, 1997.
  34. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
  35. Markov chain Monte Carlo in practice. CRC press, 1995.
  36. David Pollard. Asymptopia. Manuscript, Yale University, Dept. of Statist., New Haven, Connecticut, 2000.
  37. P Alfeld. Understanding mathematics. utah: Departemen of mathematics, 2014.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up