Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Active Third-Person Imitation Learning (2312.16365v1)

Published 27 Dec 2023 in cs.LG, cs.AI, and stat.ML

Abstract: We consider the problem of third-person imitation learning with the additional challenge that the learner must select the perspective from which they observe the expert. In our setting, each perspective provides only limited information about the expert's behavior, and the learning agent must carefully select and combine information from different perspectives to achieve competitive performance. This setting is inspired by real-world imitation learning applications, e.g., in robotics, a robot might observe a human demonstrator via camera and receive information from different perspectives depending on the camera's position. We formalize the aforementioned active third-person imitation learning problem, theoretically analyze its characteristics, and propose a generative adversarial network-based active learning approach. Empirically, we demstrate that our proposed approach can effectively learn from expert demonstrations and explore the importance of different architectural choices for the learner's performance.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (55)
  1. Pieter Abbeel and Andrew Y Ng. 2004. Apprenticeship learning via inverse reinforcement learning. In International Conference on Machine Learning (ICML).
  2. Deep Reinforcement Learning at the Edge of the Statistical Precipice. ([n. d.]), 27.
  3. Saurabh Arora and Prashant Doshi. 2021. A survey of inverse reinforcement learning: Challenges, methods and progress. Artificial Intelligence 297 (2021), 103500.
  4. Humanoid robot learning and game playing using PC-based vision. In IEEE/RSJ International Conference on Intelligent Robots and Systems, Vol. 3. IEEE, 2449–2454.
  5. Lukas Biewald. 2020. Experiment Tracking with Weights and Biases.
  6. Active Imitation Learning with Noisy Guidance. In Annual Meeting of the Association for Computational Linguistics. 2093–2105.
  7. Openai gym. arXiv preprint arXiv:1606.01540 (2016).
  8. Interactive Imitation Learning in Robotics: A Survey. Foundations and Trends® in Robotics 10, 1-2 (2022), 1–197. http://dx.doi.org/10.1561/2300000072
  9. Causal confusion in imitation learning. Advances in Neural Information Processing Systems (NeurIPS) (2019).
  10. Norman Di Palo and Edward Johns. 2020. SAFARI: Safe and active robot imitation learning with imagination. arXiv preprint arXiv:2011.09586 (2020).
  11. Power Grid Congestion Management via Topology Optimization with AlphaZero. CoRR abs/2211.05612 (2022). https://doi.org/10.48550/arXiv.2211.05612 arXiv:2211.05612
  12. Wael Farag and Zakaria Saleh. 2018. Behavior Cloning for Autonomous Driving using Convolutional Neural Networks. In 2018 International Conference on Innovation and Intelligence for Informatics, Computing, and Technologies (3ICT). 1–7. https://doi.org/10.1109/3ICT.2018.8855753
  13. Towards Third-Person Visual Imitation Learning Using Generative Adversarial Networks. In 2022 IEEE International Conference on Development and Learning (ICDL). IEEE, London, United Kingdom, 121–126. https://doi.org/10.1109/ICDL53763.2022.9962214
  14. Generative Adversarial Nets. In Advances in Neural Information Processing Systems (NeurIPS).
  15. HyperNetworks. arXiv:1609.09106 [cs] (Dec. 2016). arXiv:1609.09106 [cs]
  16. Array Programming with NumPy. Nature 585, 7825 (Sept. 2020), 357–362. https://doi.org/10.1038/s41586-020-2649-2
  17. Teaching inverse reinforcement learners via features and demonstrations. Advances in Neural Information Processing Systems (NeurIPS) 31 (2018).
  18. Optiongan: Learning joint reward-policy options using generative adversarial inverse reinforcement learning. In Conference on Artificial Intelligence (AAAI), Vol. 32.
  19. Jonathan Ho and Stefano Ermon. 2016. Generative Adversarial Imitation Learning. In Advances in Neural Information Processing Systems (NeurIPS), D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (Eds.).
  20. Deep imitation learning for 3D navigation tasks. Neural computing and applications 29 (2018), 389–404.
  21. Imitation learning: A survey of learning methods. ACM Computing Surveys (CSUR) 50, 2 (2017), 1–35.
  22. Movement imitation with nonlinear dynamical systems in humanoid robots. In International Conference on Robotics and Automation (ICRA). IEEE, 1398–1403.
  23. Active imitation learning via reduction to iid active learning. In 2012 AAAI Fall Symposium Series.
  24. Interactive Teaching Algorithms for Inverse Reinforcement Learning. In International Joint Conference on Artificial Intelligence (IJCAI). 2692–2700.
  25. VOILA: Visual-Observation-Only Imitation Learning for Autonomous Navigation. In International Conference on Robotics and Automation (ICRA). 2497–2503.
  26. Tor Lattimore and Csaba Szepesvári. 2020. Bandit algorithms. Cambridge University Press.
  27. Active Exploration for Inverse Reinforcement Learning. arXiv preprint arXiv:2207.08645 (2022).
  28. Learning human behaviors from motion capture by adversarial imitation. arXiv preprint arXiv:1707.02201 (2017).
  29. Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05957 (2018).
  30. Combining self-supervised learning and imitation for vision-based rope manipulation. In International Conference on Robotics and Automation (ICRA). IEEE, 2146–2153.
  31. Algorithms for inverse reinforcement learning.. In International Conference on Machine Learning (ICML).
  32. PyTorch: An Imperative Style, High-Performance Deep Learning Library. ([n. d.]), 12.
  33. Zero-shot visual imitation. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 2050–2053.
  34. FiLM: Visual Reasoning with a General Conditioning Layer. arXiv:1709.07871 [cs, stat]
  35. Plotly-Community. [n.d.]. Plotly/Plotly.Py: The Interactive Graphing Library for Python.
  36. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434 (2015).
  37. Stéphane Ross and Drew Bagnell. 2010. Efficient reductions for imitation learning. In International Conference on Artificial Intelligence and Statistics (AISTATS). 661–668.
  38. Stuart Russell. 1998. Learning agents for uncertain environments. In Conference on Computational Learning Theory. 101–103.
  39. Mastering atari, go, chess and shogi by planning with a learned model. Nature 588, 7839 (2020), 604–609.
  40. Trust Region Policy Optimization. In International Conference on Machine Learning (ICML). 1889–1897.
  41. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017).
  42. Time-Contrastive Networks: Self-Supervised Learning from Video. arXiv:1704.06888 [cs]
  43. Jinghuan Shang and Michael S. Ryoo. 2021. Self-Supervised Disentangled Representation Learning for Third-Person Imitation Learning. arXiv:2108.01069 [cs]
  44. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv preprint arXiv:1712.01815 (2017).
  45. Reinforcement learning in robotic applications: a comprehensive survey. Artificial Intelligence Review 55, 2 (2022), 945–990.
  46. Third Person Imitation Learning. In International Conference on Learning Representations (ICLR).
  47. Richard S Sutton and Andrew G Barto. 2018. Reinforcement learning: An introduction. MIT press.
  48. Behavioral cloning from observation. In International Joint Conference on Artificial Intelligence (IJCAI). 4950–4957.
  49. Generative Adversarial Imitation from Observation. arXiv:1807.06158 [cs, stat]
  50. Recent Advances in Imitation Learning from Observation. In International Joint Conference on Artificial Intelligence (IJCAI) (Macao, China, 2019-08). 6325–6331.
  51. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature 575, 7782 (2019), 350–354.
  52. Maximum entropy deep inverse reinforcement learning. arXiv preprint arXiv:1507.04888 (2015).
  53. XIRL: Cross-embodiment Inverse Reinforcement Learning. ([n. d.]).
  54. DeepThermal: Combustion Optimization for Thermal Power Generating Units Using Offline Reinforcement Learning. In AAAI.
  55. Sim-to-real transfer in deep reinforcement learning for robotics: a survey. In Symposium Series on Computational Intelligence (SSCI). IEEE, 737–744.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now