Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
97 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Human Demonstrations are Generalizable Knowledge for Robots (2312.02419v2)

Published 5 Dec 2023 in cs.RO

Abstract: Learning from human demonstrations is an emerging trend for designing intelligent robotic systems. However, previous methods typically regard videos as instructions, simply dividing them into action sequences for robotic repetition, which poses obstacles to generalization to diverse tasks or object instances. In this paper, we propose a different perspective, considering human demonstration videos not as mere instructions, but as a source of knowledge for robots. Motivated by this perspective and the remarkable comprehension and generalization capabilities exhibited by LLMs, we propose DigKnow, a method that DIstills Generalizable KNOWledge with a hierarchical structure. Specifically, DigKnow begins by converting human demonstration video frames into observation knowledge. This knowledge is then subjected to analysis to extract human action knowledge and further distilled into pattern knowledge compassing task and object instances, resulting in the acquisition of generalizable knowledge with a hierarchical structure. In settings with different tasks or object instances, DigKnow retrieves relevant knowledge for the current task and object instances. Subsequently, the LLM-based planner conducts planning based on the retrieved knowledge, and the policy executes actions in line with the plan to achieve the designated task. Utilizing the retrieved knowledge, we validate and rectify planning and execution outcomes, resulting in a substantial enhancement of the success rate. Experimental results across a range of tasks and scenes demonstrate the effectiveness of this approach in facilitating real-world robots to accomplish tasks with the knowledge derived from human demonstrations.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (46)
  1. Chatgpt empowered long-step robot control in various environments: A case application. arXiv preprint arXiv:2304.03893, 2023.
  2. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference on Machine Learning, pages 9118–9147. PMLR, 2022.
  3. Creative robot tool use with large language models. arXiv preprint arXiv:2310.13065, 2023.
  4. Generalizable long-horizon manipulations with large language models. arXiv preprint arXiv:2310.02264, 2023.
  5. Grid: Scene-graph-based instruction-driven robotic task planning. arXiv preprint arXiv:2309.07726, 2023.
  6. Interactive task planning with language models. arXiv preprint arXiv:2310.10645, 2023.
  7. Distilling and retrieving generalizable knowledge for robot manipulation via language corrections. arXiv preprint arXiv:2311.10678, 2023.
  8. Reflect: Summarizing robot experiences for failure explanation and correction. arXiv preprint arXiv:2306.15724, 2023.
  9. Robot learning manipulation action plans by” watching” unconstrained videos from the world wide web. In Proceedings of the AAAI conference on artificial intelligence, volume 29, 2015.
  10. Learning robot activities from first-person human videos using convolutional future regression. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 1–2, 2017.
  11. Gpt-4v (ision) for robotics: Multimodal task planning from human demonstration. arXiv preprint arXiv:2311.12015, 2023.
  12. Stefan Schaal. Learning from demonstration. Advances in neural information processing systems, 9, 1996.
  13. Recent advances in robot learning from demonstration. Annual review of control, robotics, and autonomous systems, 3:297–330, 2020.
  14. Concept2robot: Learning manipulation concepts from instructions and human demonstrations. The International Journal of Robotics Research, 40(12-14):1419–1434, 2021.
  15. Dexmv: Imitation learning for dexterous manipulation from human videos. In European Conference on Computer Vision, pages 570–587. Springer, 2022.
  16. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
  17. Gpt-neox-20b: An open-source autoregressive language model, 2022.
  18. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
  19. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
  20. Falcon-40b: an open large language model with state-of-the-art performance. Findings of the Association for Computational Linguistics: ACL, 2023:10755–10773, 2023.
  21. Llm-planner: Few-shot grounded planning for embodied agents with large language models, 2023.
  22. Text2motion: from natural language instructions to feasible plans. Autonomous Robots, 47(8):1345–1365, November 2023.
  23. Code as policies: Language model programs for embodied control. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 9493–9500. IEEE, 2023.
  24. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608, 2022.
  25. Perceiver-actor: A multi-task transformer for robotic manipulation. In Conference on Robot Learning, pages 785–799. PMLR, 2023.
  26. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818, 2023.
  27. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817, 2022.
  28. Vision-language foundation models as effective robot imitators. arXiv preprint arXiv:2311.01378, 2023.
  29. Mutex: Learning unified policies from multimodal task specifications. arXiv preprint arXiv:2309.14320, 2023.
  30. Robot task planning and situation handling in open worlds. arXiv preprint arXiv:2210.01287, 2022.
  31. Task and motion planning with large language models for object rearrangement. arXiv preprint arXiv:2303.06247, 2023.
  32. Do as i can, not as i say: Grounding language in robotic affordances. In Conference on Robot Learning, pages 287–318. PMLR, 2023.
  33. Progprompt: Generating situated robot task plans using large language models. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 11523–11530. IEEE, 2023.
  34. Robot learning from demonstration by constructing skill trees. The International Journal of Robotics Research, 31(3):360–375, 2012.
  35. A survey of robot learning from demonstration. Robotics and autonomous systems, 57(5):469–483, 2009.
  36. Learning generalizable robotic reward functions from” in-the-wild” human videos. arXiv preprint arXiv:2103.16817, 2021.
  37. Bc-z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learning, pages 991–1002. PMLR, 2022.
  38. Avid: Learning multi-stage tasks via pixel-level translation of human videos. arXiv preprint arXiv:1912.04443, 2019.
  39. Learning by watching: Physical imitation of manipulation skills from human videos. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7827–7834. IEEE, 2021.
  40. Transformers for one-shot visual imitation. In Conference on Robot Learning, pages 2071–2084. PMLR, 2021.
  41. R3m: A universal visual representation for robot manipulation. arXiv preprint arXiv:2203.12601, 2022.
  42. A learning-from-observation framework: One-shot robot teaching for grasp-manipulation-release household operations. In 2021 IEEE/SICE International Symposium on System Integration (SII), pages 461–466. IEEE, 2021.
  43. Segment anything. arXiv:2304.02643, 2023.
  44. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023.
  45. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023.
  46. OpenAI. Gpt-4 technical report, 2023.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (10)
  1. Guangyan Chen (5 papers)
  2. Te Cui (4 papers)
  3. Tianxing Zhou (2 papers)
  4. Zicai Peng (2 papers)
  5. Mengxiao Hu (4 papers)
  6. Meiling Wang (14 papers)
  7. Yi Yang (856 papers)
  8. Yufeng Yue (28 papers)
  9. Haoyang Lu (3 papers)
  10. Haizhou Li (286 papers)
Citations (5)
View on Semantic Scholar

Summary

  • The paper introduces DigKnow, a method that distills human demonstration videos into hierarchical observation, action, and pattern knowledge.
  • It leverages keyframe analysis and large language models to enable robots to plan and correct actions for novel tasks effectively.
  • Experiments show that DigKnow enhances robot generalization across varied contexts, indicating promising improvements in adaptive task performance.

Introduction

The integration of human demonstration videos into robotic system learning represents an innovative approach within the field of robotics. Instead of simply replicating actions, human demonstrations are now viewed as a repository of knowledge from which robots can draw to perform various tasks. This process entails converting human actions observed in videos into a structured form of knowledge that robots can understand and adapt to different situations.

Knowledge Distillation Approach

A new methodology called DigKnow has been developed to process and sublimate human demonstration videos into hierarchical knowledge, which can be accessed and used by robots. The process begins with the analysis of video frames to extract 'observation knowledge', which includes understanding the spatial relationships within the scene. Subsequent stages involve generating 'action knowledge' through keyframe analysis and, ultimately, distilling this information into 'pattern knowledge', which divides into task-specific and object-specific insights. Importantly, this hierarchical knowledge enables robots to better generalize and adapt to new environments or tasks.

Knowledge Retrieval and Correction

When facing new tasks or objects, the robots tap into the stored knowledge to formulate a plan that aligns with the current requirements. This planning process is facilitated by employing LLMs, which assist in interpreting and integrating the relevant knowledge to create actionable sequences. Moreover, DigKnow features a correction component that leverages the gathered knowledge to validate and correct action plans and executions, thereby optimizing the robot's performance and adaptability.

Experiments and Results

The efficacy of DigKnow has been assessed through real-world experiments using diverse tasks and environmental setups. These assessments demonstrate the system's proficiency in generalizing skills derived from human demonstrations across various contexts. It should be noted, however, that the current scope of testing is rather limited. Future expansions of experimental setups are planned to comprehensively validate DigKnow's performance.

Conclusion

DigKnow represents a significant advance in robot learning methodologies by leveraging human demonstrations as a rich source of knowledge, rather than mere sequential instructions. Its hierarchical knowledge structure enables robots to retrieve relevant information for novel tasks and objects, while its correction mechanisms help in achieving a high success rate even in unfamiliar scenarios. If further testing confirms these early results, DigKnow holds the potential to greatly enhance a robot's ability to perform complex tasks informed by human experiences.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown