Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
153 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

A Kernel Perspective on Behavioural Metrics for Markov Decision Processes (2310.19804v1)

Published 5 Oct 2023 in cs.LG and cs.AI

Abstract: Behavioural metrics have been shown to be an effective mechanism for constructing representations in reinforcement learning. We present a novel perspective on behavioural metrics for Markov decision processes via the use of positive definite kernels. We leverage this new perspective to define a new metric that is provably equivalent to the recently introduced MICo distance (Castro et al., 2021). The kernel perspective further enables us to provide new theoretical results, which has so far eluded prior work. These include bounding value function differences by means of our metric, and the demonstration that our metric can be provably embedded into a finite-dimensional Euclidean space with low distortion error. These are two crucial properties when using behavioural metrics for reinforcement learning representations. We complement our theory with strong empirical results that demonstrate the effectiveness of these methods in practice.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (79)
  1. Contrastive behavioral similarity embeddings for generalization in reinforcement learning. In In Proceedings of the Ninth International Conference on Learning Representations, 2021a.
  2. Deep reinforcement learning at the edge of the statistical precipice. In Advances in Neural Information Processing Systems, 2021b.
  3. Vector-valued Laplace transforms and Cauchy problems. Springer-Verlag, Jan 2001.
  4. Nachman Aronszajn. Theory of reproducing kernels. Transactions of the American mathematical society, 68(3):337–404, 1950.
  5. Leemon C. Baird. Residual algorithms: Reinforcement learning with function approximation. In Proceedings of the International Conference on Machine Learning, 1995.
  6. Practical kernel-based reinforcement learning. The Journal of Machine Learning Research, 17(1):2372–2441, 2016.
  7. The Arcade Learning Environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253–279, June 2013.
  8. A geometric perspective on optimal representations for reinforcement learning. In Advances in Neural Information Processing Systems, 2019.
  9. Neuro-dynamic programming. Athena Scientific, 1996.
  10. Bisimulation for labelled Markov processes. In Proceedings of IEEE Symposium On Logic In Computer Science, 1997.
  11. Jax: composable transformations of python+ numpy programs. 2018.
  12. Pablo Samuel Castro. Scalable methods for computing state similarity in deterministic Markov decision processes. In Proceedings of the AAAI Conference on Artificial Intelligence, 2020.
  13. Dopamine: A Research Framework for Deep Reinforcement Learning. arXiv, 2018.
  14. MICo: Learning improved representations via sampling-based state similarity for Markov decision processes. In Advances in Neural Information Processing Systems, 2021.
  15. On-the-fly algorithms for bisimulation metrics. In Proceedings of the International Conference on Quantitative Evaluation of Systems, 2012.
  16. Implicit quantile networks for distributional reinforcement learning. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1096–1105. PMLR, 2018a.
  17. Distributional reinforcement learning with quantile regression. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018b.
  18. Peter Dayan. Improving generalization for temporal difference learning: The successor representation. Neural Computation, 5(4):613–624, 1993.
  19. Bisimulation for labeled Markov processes. Information and Computation, 179(2):163–193, Dec 2002.
  20. Metrics for labeled Markov systems. In Proceedings of the International Conference on Concurrency Theory, 1999.
  21. Geometry of cuts and metrics, volume 2. Springer, 1997.
  22. Kernel-based reinforcement learning: A finite-time analysis. In Proceedings of the International Conference on Machine Learning, 2021.
  23. Regularized policy iteration with nonparametric function spaces. The Journal of Machine Learning Research, 17:1–66, January 2016.
  24. Metrics for finite Markov decision processes. In Proceedings of the Conference on Uncertainty in Artificial Intelligence, 2004.
  25. Methods for computing state similarity in Markov decision processes. In Conference on Uncertainty in Artificial Intelligence (UAI), 2006.
  26. Bisimulation metrics for continuous Markov decision processes. SIAM Journal on Computing, 40(6):1662–1714, 2011.
  27. Learning visual feature spaces for robotic manipulation with deep spatial autoencoders. arXiv, 2015.
  28. DeepMDP: Learning continuous latent space models for representation learning. In Proceedings of the International Conference on Machine Learning, 2019.
  29. Corrado Gini. Variabilità e mutabilità: contributo allo studio delle distribuzioni e delle relazioni statistiche. Studi economico-giuridici pubblicati per cura della facoltà di Giurisprudenza della R. Università di Cagliari. Tipogr. di P. Cuppini, 1912.
  30. Equivalence notions and model minimization in Markov decision processes. Artificial Intelligence, 147(1-2):163–223, 2003.
  31. A kernel two-sample test. Journal of Machine Learning Research, 13(25):723–773, 2012.
  32. C. Guilbart. Produits scalaires sur l’espace des mesures. Annales de l’I.H.P. Probabilités et statistiques, 15(4):333–354, 1979.
  33. Learning latent dynamics for planning from pixels. In Proceedings of the International Conference on Machine Learning, 2019.
  34. Bisimulation makes analogies in goal-conditioned reinforcement learning. In Proceedings of the International Conference on Machine Learning, 2022.
  35. Array programming with numpy. Nature, 585(7825):357–362, 2020.
  36. Rainbow: Combining Improvements in Deep Reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018.
  37. John D Hunter. Matplotlib: A 2d graphics environment. Computing in science & engineering, 9(03):90–95, 2007.
  38. Reinforcement learning with unsupervised auxiliary tasks. In Proceedings of the International Conference on Learning Representations, 2017.
  39. Extensions of Lipschitz maps into a Hilbert space. Contemporary Mathematics, 26:189–206, 01 1984.
  40. On a space of totally additive functions. Vestnik Leningrad. Univ, 13(7):52–59, 1958.
  41. Towards robust bisimulation metric learning. Advances in Neural Information Processing Systems, 2021.
  42. Approximate policy iteration with bisimulation metrics. Transactions on Machine Learning Research, 2022.
  43. Value function approximation in reinforcement learning using the Fourier basis. In Proceedings of the AAAI Conference on Artificial Intelligence, 2011.
  44. Policy evaluation in continuous mdps with efficient kernelized gradient temporal difference. IEEE Trans. Autom. Control., 66(4):1856–1863, 2021.
  45. Deep auto-encoder neural networks in reinforcement learning. In Proceedings of the International Joint Conference on Neural Networks, 2010.
  46. Bisimulation through probablistic testing. Information and Computation, 94:1–28, 1991.
  47. Metrics and continuity in reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
  48. Compressed conditional mean embeddings for model-based reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 2016.
  49. Adaptive auxiliary task weighting for reinforcement learning. In Advances in Neural Information Processing Systems, 2019.
  50. Proto-value functions: A Laplacian framework for learning representation and control in Markov decision processes. Journal of Machine Learning Research, 8:2169–2231, Dec 2007.
  51. Jiri Matousek. Lectures on discrete geometry, volume 212. Springer Science & Business Media, 2013.
  52. Stephen G. Matthews. Partial metric topology. Annals of the New York Academy of Sciences, 728(1):183–197, 1994.
  53. Robert Milner. A Calculus for Communicating Systems, volume 92 of Lecture Notes in Computer Science. Springer-Verlag, 1980.
  54. Human-level control through deep reinforcement learning. Nature, 518(7540):529–533, 2015.
  55. Alfred Müller. Integral probability metrics and their generating classes of functions. Advances in Applied Probability, 29(2):429–443, 1997. ISSN 00018678.
  56. Travis E. Oliphant. Python for scientific computing. Computing in Science & Engineering, 9(3):10–20, 2007. doi: 10.1109/MCSE.2007.58.
  57. Kernel-based reinforcement learning. Machine Learning, 49(2–3):161–178, 2002.
  58. Prakash Panangaden. Labelled Markov processes. Imperial College Press, 2009.
  59. David Park. Title unknown. Slides for Bad Honnef Workshop on Semantics of Concurrency, 1981.
  60. Computational optimal transport. Foundations and Trends® in Machine Learning, 11(5-6):355–607, 2019.
  61. The method of distances in the theory of probability and statistics. Springer-Verlag, 2013.
  62. Frigyes Riesz. Sur une espèce de Géométrie analytique des systèmes de fonctions sommables. Gauthier-Villars, 1907.
  63. A Stochastic Approximation Method. The Annals of Mathematical Statistics, 22(3):400 – 407, 1951.
  64. Walter Rudin. Functional Analysis. Tata McGraw-Hill, 1974.
  65. Isaac J. Schoenberg. Remarks to Maurice Fréchet’s article“Sur la definition axiomatique d’une classe d’espace distances vectoriellement applicable sur l’espace de Hilbert”. Annals of Mathematics, pp.  724–732, 1935.
  66. Learning with kernels: Support vector machines, regularization, optimization, and beyond. The MIT Press, 2018.
  67. Equivalence of distance-based and RKHS-based statistics in hypothesis testing. The Annals of Statistics, 41(5):2263–2291, 2013.
  68. Loss is its own reward: Self-supervision for reinforcement learning. In Proceedings of the International Conference on Learning Representations (Workshop Track), 2017.
  69. Reinforcement Learning: An Introduction. The MIT Press, 2018.
  70. DeepMind control suite. arXiv, 2018.
  71. Franck van Breugel and James Worrell. Towards quantitative verification of probabilistic systems. In Proceedings of the International Colloquium on Automata, Languages and Programming, July 2001.
  72. Python reference manual. Centrum voor Wiskunde en Informatica Amsterdam, 1995.
  73. Munchausen reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS 2020), 2020.
  74. Cédric Villani. Optimal transport: Old and new, volume 338. Springer Science & Business Media, 2008.
  75. Provably efficient reinforcement learning with kernel and neural function approximations. In Advances in Neural Information Processing Systems, 2020.
  76. Improving sample efficiency in model-free reinforcement learning from images. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
  77. Shlomo Yitzhaki. Gini’s mean difference: A superior measure of variability for non-normal distributions. Metron - International Journal of Statistics, LXI(2):285–316, 2003.
  78. Invariant representations for reinforcement learning without reconstruction. In Proceedings of the International Conference on Learning Representations, 2021.
  79. Szymon Łukaszyk. A new concept of probability metric and its applications in approximation of scattered data sets. Computational Mechanics, 33:299–304, 03 2004.
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now