Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
60 tokens/sec
GPT-4o
12 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Discovering Object-Centric Generalized Value Functions From Pixels (2304.13892v2)

Published 27 Apr 2023 in cs.LG and cs.AI

Abstract: Deep Reinforcement Learning has shown significant progress in extracting useful representations from high-dimensional inputs albeit using hand-crafted auxiliary tasks and pseudo rewards. Automatically learning such representations in an object-centric manner geared towards control and fast adaptation remains an open research problem. In this paper, we introduce a method that tries to discover meaningful features from objects, translating them to temporally coherent "question" functions and leveraging the subsequent learned general value functions for control. We compare our approach with state-of-the-art techniques alongside other ablations and show competitive performance in both stationary and non-stationary settings. Finally, we also investigate the discovered general value functions and through qualitative analysis show that the learned representations are not only interpretable but also, centered around objects that are invariant to changes across tasks facilitating fast adaptation.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (32)
  1. Layer normalization, 2016. URL https://arxiv.org/abs/1607.06450.
  2. Minimalistic gridworld environment for openai gym. https://github.com/maximecb/gym-minigrid, 2018.
  3. Leveraging procedural generation to benchmark reinforcement learning, 2019. URL https://arxiv.org/abs/1912.01588.
  4. Learning to reach goals without reinforcement learning. 2019.
  5. Bootstrap latent-predictive representations for multitask reinforcement learning. arXiv preprint arXiv:2004.14646, 2020.
  6. Reinforcement learning with unsupervised auxiliary tasks, 2016.
  7. Population based training of neural networks. CoRR, abs/1711.09846, 2017. URL http://arxiv.org/abs/1711.09846.
  8. Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on Robot Learning, pp.  651–673. PMLR, 2018.
  9. What should i know? using meta-gradient descent for predictive feature discovery in a single stream of experience, 2022. URL https://arxiv.org/abs/2206.06485.
  10. Curl: Contrastive unsupervised representations for reinforcement learning. In International Conference on Machine Learning, pp. 5639–5650. PMLR, 2020.
  11. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):1334–1373, 2016.
  12. Contrastive explanations for reinforcement learning via embedded self predictions. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=Ud3DSz72nYR.
  13. Object-centric learning with slot attention, 2020. URL https://arxiv.org/abs/2006.15055.
  14. Human-level control through deep reinforcement learning. Nature, 518(7540):529–533, February 2015. ISSN 00280836. URL http://dx.doi.org/10.1038/nature14236.
  15. Visual reinforcement learning with imagined goals. Advances in neural information processing systems, 31, 2018.
  16. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018.
  17. Planning from pixels using inverse dynamics models. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=V6BjBgku7Ro.
  18. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pp. 2778–2787. PMLR, 2017.
  19. Skew-fit: State-covering self-supervised reinforcement learning. arXiv preprint arXiv:1903.03698, 2019.
  20. Data-efficient reinforcement learning with self-predictive representations. arXiv preprint arXiv:2007.05929, 2020.
  21. Data-efficient reinforcement learning with self-predictive representations. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=uCQfPZwRaUu.
  22. Loss is its own reward: Self-supervision for reinforcement learning. arXiv preprint arXiv:1612.07307, 2016.
  23. Reinforcement Learning: An Introduction. A Bradford Book, Cambridge, MA, USA, 2018a. ISBN 0262039249.
  24. Reinforcement learning: An introduction. MIT press, 2018b.
  25. Horde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction. In The 10th International Conference on Autonomous Agents and Multiagent Systems - Volume 2, AAMAS ’11, pp.  761–768, Richland, SC, 2011. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 0982657161.
  26. Deep reinforcement learning with double q-learning, 2015. URL https://arxiv.org/abs/1509.06461.
  27. A perspective on objects and systematic generalization in model-based rl. arXiv preprint arXiv:1906.01035, 2019.
  28. Entity abstraction in visual model-based reinforcement learning. In Conference on Robot Learning, pp.  1439–1456. PMLR, 2020.
  29. Discovery of useful questions as auxiliary tasks, 2019.
  30. Unsupervised control through non-parametric discriminative rewards. arXiv preprint arXiv:1811.11359, 2018.
  31. Cobra: Data-efficient model-based rl through unsupervised object discovery and curiosity-driven exploration. arXiv preprint arXiv:1905.09275, 2019.
  32. Self-supervised visual reinforcement learning with object-centric representations. arXiv preprint arXiv:2011.14381, 2020.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now