Papers
Topics
Authors
Recent
Detailed Answer
Quick Answer
Concise responses based on abstracts only
Detailed Answer
Well-researched responses based on abstracts and relevant paper content.
Custom Instructions Pro
Preferences or requirements that you'd like Emergent Mind to consider when generating responses
Gemini 2.5 Flash
Gemini 2.5 Flash 71 tok/s
Gemini 2.5 Pro 52 tok/s Pro
GPT-5 Medium 18 tok/s Pro
GPT-5 High 15 tok/s Pro
GPT-4o 101 tok/s Pro
Kimi K2 196 tok/s Pro
GPT OSS 120B 467 tok/s Pro
Claude Sonnet 4 37 tok/s Pro
2000 character limit reached

Never-Ending Behavior-Cloning Agent for Robotic Manipulation (2403.00336v2)

Published 1 Mar 2024 in cs.RO and cs.AI

Abstract: Relying on multi-modal observations, embodied robots could perform multiple robotic manipulation tasks in unstructured real-world environments. However, most language-conditioned behavior-cloning agents still face existing long-standing challenges, i.e., 3D scene representation and human-level task learning, when adapting into new sequential tasks in practical scenarios. We here investigate these above challenges with NBAgent in embodied robots, a pioneering language-conditioned Never-ending Behavior-cloning Agent. It can continually learn observation knowledge of novel 3D scene semantics and robot manipulation skills from skill-shared and skill-specific attributes, respectively. Specifically, we propose a skill-sharedsemantic rendering module and a skill-shared representation distillation module to effectively learn 3D scene semantics from skill-shared attribute, further tackling 3D scene representation overlooking. Meanwhile, we establish a skill-specific evolving planner to perform manipulation knowledge decoupling, which can continually embed novel skill-specific knowledge like human from latent and low-rank space. Finally, we design a never-ending embodied robot manipulation benchmark, and expensive experiments demonstrate the significant performance of our method. Visual results, code, and dataset are provided at: https://neragent.github.io.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (52)
  1. Few-shot continual active learning by a robot. In NeurIPS, volume 35, pp.  30612–30624, 2022.
  2. Cbcl-pr: A cognitively inspired model for class-incremental learning in robotics. IEEE Transactions on Cognitive and Developmental Systems, 2023.
  3. Continual learning through human-robot interaction–human perceptions of a continual learning robot in repeated interactions. arXiv preprint arXiv:2305.16332, 2023.
  4. Rainbow memory: Continual learning with a memory of diverse samples. In CVPR, pp.  8218–8227, 2021.
  5. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817, 2022.
  6. Do as i can, not as i say: Grounding language in robotic affordances. In CoRL, pp.  287–318. PMLR, 2023.
  7. On tiny episodic memories in continual learning. arXiv preprint arXiv:1902.10486, 2019.
  8. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1–113, 2023.
  9. Kernel continual learning. In ICML, pp.  2621–2631. PMLR, 2021.
  10. Heterogeneous forgetting compensation for class-incremental learning. In ICCV, pp.  11742–11751, 2023.
  11. Podnet: Pooled outputs distillation for small-tasks incremental learning. In ECCV, pp.  86–102. Springer, 2020.
  12. Reinforcement learning with neural radiance fields. In NeurIPS, volume 35, pp.  16931–16945, 2022.
  13. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378, 2023.
  14. Cril: Continual robot imitation learning via generative and prediction model. In IROS, pp.  6747–5754. IEEE, 2021.
  15. Ifor: Iterative flow minimization for robotic object rearrangement. In CVPR, pp.  14787–14797, 2022.
  16. Rvt: Robotic view transformer for 3d object manipulation. arXiv preprint arXiv:2306.14896, 2023.
  17. Continual robot learning using self-supervised task inference. IEEE Transactions on Cognitive and Developmental Systems, 2023.
  18. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.
  19. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608, 2022.
  20. Perceiver io: A general architecture for structured inputs & outputs. arXiv preprint arXiv:2107.14795, 2021.
  21. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5(2):3019–3026, 2020.
  22. Coarse-to-fine q-attention: Efficient learning for visual robotic manipulation via discretisation. In CVPR, pp.  13739–13748, 2022.
  23. Vima: General robot manipulation with multimodal prompts. arXiv, 2022.
  24. Continual learning with node-importance based adaptive group sparse regularization. In NeurIPS, volume 33, pp.  3647–3658, 2020.
  25. Langley, P. Crafting papers on machine learning. In Langley, P. (ed.), ICML, pp.  1207–1216, Stanford, CA, 2000. Morgan Kaufmann.
  26. Your diffusion model is secretly a zero-shot classifier. In ICCV, pp.  2206–2217, October 2023.
  27. Continual few-shot intent detection. In COLING, pp.  333–343, 2022.
  28. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935–2947, 2017.
  29. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99–106, 2021.
  30. Learning transferable visual models from natural language supervision. In ICML, pp.  8748–8763. PMLR, 2021.
  31. icarl: Incremental classifier and representation learning. In CVPR, pp.  2001–2010, 2017.
  32. V-rep: A versatile and scalable robot simulation framework. In IROS, pp.  1321–1326. IEEE, 2013.
  33. High-resolution image synthesis with latent diffusion models. In CVPR, pp.  10684–10695, 2022.
  34. Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action. In CoRL, pp.  492–504. PMLR, 2023.
  35. Snerl: Semantic-aware neural radiance fields for reinforcement learning. In ICML. PMLR, 2023.
  36. Cliport: What and where pathways for robotic manipulation. In CoRL, pp.  894–906. PMLR, 2022.
  37. Perceiver-actor: A multi-task transformer for robotic manipulation. In CoRL, pp.  785–799. PMLR, 2023.
  38. Create your world: Lifelong text-to-image diffusion. arXiv preprint arXiv:2309.04430, 2023.
  39. Exploring example influence in continual learning. In NeurIPS, volume 35, pp.  27075–27086, 2022.
  40. Gcr: Gradient coreset based replay buffer selection for continual learning. In CVPR, pp.  99–108, 2022.
  41. Bring evanescent representations to life in lifelong class incremental learning. In CVPR, pp.  16732–16741, 2022.
  42. Lotus: Continual imitation learning for robot manipulation through unsupervised skill discovery. arXiv preprint arXiv:2311.02058, 2023.
  43. Coscl: Cooperation of small continual learners is stronger than a big one. In ECCV, pp.  254–271. Springer, 2022.
  44. A comprehensive survey of continual learning: Theory, method and application. arXiv preprint arXiv:2302.00487, 2023.
  45. Incremental learning via rate reduction. In CVPR, pp.  1125–1133, 2021.
  46. Incremental learning using conditional adversarial networks. In ICCV, pp.  6619–6628, 2019.
  47. Open-vocabulary panoptic segmentation with text-to-image diffusion models. In CVPR, pp.  2955–2966, 2023.
  48. Continual object detection via prototypical task correlation guided gating mechanism. In CVPR, pp.  9255–9264, 2022.
  49. Large batch optimization for deep learning: Training bert in 76 minutes. arXiv preprint arXiv:1904.00962, 2019.
  50. pixelnerf: Neural radiance fields from one or few images. In CVPR, pp.  4578–4587, 2021.
  51. Gnfactor: Multi-task real robot learning with generalizable neural feature fields. In CoRL, pp.  284–301. PMLR, 2023.
  52. Rt-2: Vision-language-action models transfer web knowledge to robotic control. In CoRL, pp.  2165–2183. PMLR, 2023.
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Dice Question Streamline Icon: https://streamlinehq.com

Follow-Up Questions

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets