Proper Laplacian Representation Learning
Abstract: The ability to learn good representations of states is essential for solving large reinforcement learning problems, where exploration, generalization, and transfer are particularly challenging. The Laplacian representation is a promising approach to address these problems by inducing informative state encoding and intrinsic rewards for temporally-extended action discovery and reward shaping. To obtain the Laplacian representation one needs to compute the eigensystem of the graph Laplacian, which is often approximated through optimization objectives compatible with deep learning approaches. These approximations, however, depend on hyperparameters that are impossible to tune efficiently, converge to arbitrary rotations of the desired eigenvectors, and are unable to accurately recover the corresponding eigenvalues. In this paper we introduce a theoretically sound objective and corresponding optimization algorithm for approximating the Laplacian representation. Our approach naturally recovers both the true eigenvectors and eigenvalues while eliminating the hyperparameter dependence of previous approximations. We provide theoretical guarantees for our method and we show that those results translate empirically into robust learning across multiple environments.
- Carmen Chicone. Ordinary Differential Equations with Applications. Springer, 2006.
- Peter Dayan. Improving Generalization for Temporal Difference Learning: The Successor Representation. Neural Computation, 5(4):613–624, 1993.
- Deep Laplacian-based Options for Temporally-Extended Exploration. In International Conference on Machine Learning (ICML), 2023.
- Yehuda Koren. On Spectral Graph Drawing. In International Computing and Combinatorics Conference (COCOON), 2003.
- On the Generalization of Representations in Reinforcement Learning. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2022.
- A Laplacian Framework for Option Discovery in Reinforcement Learning. In International Conference on Machine Learning (ICML), 2017.
- Eigenoption Discovery through the Deep Successor Representation. In International Conference on Learning Representations (ICLR), 2018.
- Temporal Abstraction in Reinforcement Learning with the Successor Representation. Journal of Machine Learning Research, 24(80):1–69, 2023.
- Sridhar Mahadevan. Proto-value Functions: Developmental Reinforcement Learning. In International Conference on Machine Learning (ICML), 2005.
- Proto-value Functions: A Laplacian Framework for Learning Representation and Control in Markov Decision Processes. Journal of Machine Learning Research, 8(10):2169–2231, 2007.
- On Gradient-Based Learning in Continuous Games. SIAM Journal on Mathematics of Data Science, 2(1):103–131, 2020.
- Numerical Optimization. Spinger, 2006.
- Spectral Inference Networks: Unifying Deep and Spectral Learning. In International Conference on Learning Representations (ICLR), 2019.
- Shankar Sastry. Nonlinear Systems: Analysis, Stability, and Control. Springer, 2013.
- Design Principles of the Hippocampal Cognitive Map. Advances in Neural Information Processing Systems (NeurIPS), 2014.
- No More Pesky Hyperparameters: Offline Hyperparameter Tuning for RL. Transactions on Machine Learning Research, 2022, 2022.
- Towards Better Laplacian Representation in Reinforcement Learning with Generalized Graph Drawing. In International Conference on Machine Learning (ICML), 2021.
- Reachability-Aware Laplacian Representation in Reinforcement Learning. In International Conference on Machine Learning (ICML), 2023.
- The Laplacian in RL: Learning Representations with Efficient Approximations. In International Conference on Learning Representations (ICLR), 2019.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.
