Papers
Topics
Authors
Recent
Assistant
AI Research Assistant
Well-researched responses based on relevant abstracts and paper content.
Custom Instructions Pro
Preferences or requirements that you'd like Emergent Mind to consider when generating responses.
Gemini 2.5 Flash
Gemini 2.5 Flash 134 tok/s
Gemini 2.5 Pro 41 tok/s Pro
GPT-5 Medium 26 tok/s Pro
GPT-5 High 27 tok/s Pro
GPT-4o 100 tok/s Pro
Kimi K2 204 tok/s Pro
GPT OSS 120B 433 tok/s Pro
Claude Sonnet 4.5 37 tok/s Pro
2000 character limit reached

R3L: Relative Representations for Reinforcement Learning (2404.12917v3)

Published 19 Apr 2024 in cs.LG, cs.AI, and cs.CV

Abstract: Visual Reinforcement Learning is a popular and powerful framework that takes full advantage of the Deep Learning breakthrough. It is known that variations in input domains (e.g., different panorama colors due to seasonal changes) or task domains (e.g., altering the target speed of a car) can disrupt agent performance, necessitating new training for each variation. Recent advancements in the field of representation learning have demonstrated the possibility of combining components from different neural networks to create new models in a zero-shot fashion. In this paper, we build upon relative representations, a framework that maps encoder embeddings to a universal space. We adapt this framework to the Visual Reinforcement Learning setting, allowing to combine agents components to create new agents capable of effectively handling novel visual-task pairs not encountered during training. Our findings highlight the potential for model reuse, significantly reducing the need for retraining and, consequently, the time and computational resources required.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (34)
  1. Revisiting model stitching to compare neural representations. In Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp.  225–236, 2021. URL https://proceedings.neurips.cc/paper/2021/hash/01ded4259d101feb739b06c399e9cd9c-Abstract.html.
  2. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253–279, 2013.
  3. From bricks to bridges: Product of invariances to enhance latent space communication, 2023.
  4. Quantifying generalization in reinforcement learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp.  1282–1289. PMLR, 09–15 Jun 2019. URL https://proceedings.mlr.press/v97/cobbe19a.html.
  5. Similarity and matching of neural network representations. ArXiv preprint, abs/2110.14633, 2021. URL https://arxiv.org/abs/2110.14633.
  6. Learning modular neural network policies for multi-task and multi-robot transfer. In 2017 IEEE international conference on robotics and automation (ICRA), pp.  2169–2176. IEEE, 2017.
  7. Bisimulation metrics are optimal value functions. In UAI, pp.  210–219, 2014.
  8. Towards reusable network components by learning compatible representations. AAAI, 35(9):7620–7629, 2021.
  9. Self-supervised policy adaptation during deployment. arXiv preprint arXiv:2007.04309, 2020.
  10. Stabilizing deep q-learning with convnets and vision transformers under data augmentation. In Conference on Neural Information Processing Systems, 2021.
  11. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23(274):1–18, 2022. URL http://jmlr.org/papers/v23/21-1342.html.
  12. Zero-shot reinforcement learning via function encoders. arXiv preprint arXiv:2401.17173, 2024.
  13. Policy stitching: Learning transferable robot policies. arXiv preprint arXiv:2309.13753, 2023.
  14. Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on Robot Learning, pp.  651–673. PMLR, 2018.
  15. Oleg Klimov. Carracing-v0. URL https://gym. openai. com/envs/CarRacing-v0, 2016.
  16. Curl: Contrastive unsupervised representations for reinforcement learning. In International Conference on Machine Learning, pp.  5639–5650. PMLR, 2020a.
  17. Reinforcement learning with augmented data. Advances in neural information processing systems, 33:19884–19895, 2020b.
  18. Network randomization: A simple technique for generalization in deep reinforcement learning. In ICLR, 2020.
  19. Understanding image representations by measuring their equivariance and equivalence. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pp.  991–999. IEEE Computer Society, 2015. doi: 10.1109/CVPR.2015.7298701. URL https://doi.org/10.1109/CVPR.2015.7298701.
  20. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):1334–1373, 2016.
  21. Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. The International journal of robotics research, 37(4-5):421–436, 2018.
  22. Human-level control through deep reinforcement learning. nature, 518(7540):529–533, 2015.
  23. Relative representations enable zero-shot latent space communication. In International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=SrC-nwieGJ.
  24. ASIF: Coupled Data Turns Unimodal Models to Multimodal Without Training. ArXiv preprint, abs/2210.01738, 2022. URL https://arxiv.org/abs/2210.01738.
  25. Automatic data augmentation for generalization in reinforcement learning. Advances in Neural Information Processing Systems, 34:5402–5415, 2021.
  26. A survey on image data augmentation for deep learning. Journal of big data, 6(1):1–48, 2019.
  27. Observational overfitting in reinforcement learning. arXiv preprint arXiv:1912.02975, 2019.
  28. Reinforcement Learning: An Introduction. The MIT Press, Cambridge, MA, 1998.
  29. Learning and predicting photonic responses of plasmonic nanoparticle assemblies via dual variational autoencoders. ArXiv preprint, abs/2208.03861, 2022. URL https://arxiv.org/abs/2208.03861.
  30. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In International conference on learning representations, 2020.
  31. Invariance through latent alignment. arXiv preprint arXiv:2112.08526, 2021.
  32. Don’t touch what matters: Task-aware lipschitz data augmentation for visual reinforcement learning. arXiv preprint arXiv:2202.09982, 2022.
  33. Natural environment benchmarks for reinforcement learning. arXiv preprint arXiv:1811.06032, 2018.
  34. Learning invariant representations for reinforcement learning without reconstruction. arXiv preprint arXiv:2006.10742, 2020.
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Dice Question Streamline Icon: https://streamlinehq.com

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
Lightbulb Streamline Icon: https://streamlinehq.com

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets

This paper has been mentioned in 1 tweet and received 1 like.

Upgrade to Pro to view all of the tweets about this paper:

Start a free 7-day Pro trial