Papers
Topics
Authors
Recent
Search
2000 character limit reached

On the Theory of Risk-Aware Agents: Bridging Actor-Critic and Economics

Published 30 Oct 2023 in cs.LG | (2310.19527v3)

Abstract: Risk-aware Reinforcement Learning (RL) algorithms like SAC and TD3 were shown empirically to outperform their risk-neutral counterparts in a variety of continuous-action tasks. However, the theoretical basis for the pessimistic objectives these algorithms employ remains unestablished, raising questions about the specific class of policies they are implementing. In this work, we apply the expected utility hypothesis, a fundamental concept in economics, to illustrate that both risk-neutral and risk-aware RL goals can be interpreted through expected utility maximization using an exponential utility function. This approach reveals that risk-aware policies effectively maximize value certainty equivalent, aligning them with conventional decision theory principles. Furthermore, we propose Dual Actor-Critic (DAC). DAC is a risk-aware, model-free algorithm that features two distinct actor networks: a pessimistic actor for temporal-difference learning and an optimistic actor for exploration. Our evaluations of DAC across various locomotion and manipulation tasks demonstrate improvements in sample efficiency and final performance. Remarkably, DAC, while requiring significantly less computational resources, matches the performance of leading model-based methods in the complex dog and humanoid domains.

Authors (2)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (76)
  1. Maximum a posteriori policy optimisation. In International Conference on Learning Representations, 2018.
  2. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 34:29304–29320, 2021.
  3. Amari, S.-I. Natural gradient works efficiently in learning. Neural computation, 10(2):251–276, 1998.
  4. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47:235–256, 2002.
  5. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
  6. Ready policy one: World building through active learning. In International Conference on Machine Learning, pp.  591–601. PMLR, 2020.
  7. Efficient online reinforcement learning with offline data. 2023.
  8. Distributed distributional deterministic policy gradients. In International Conference on Learning Representations, 2018.
  9. Learning pessimism for reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp.  6971–6979, 2023.
  10. Ucb exploration via q-ensembles. arXiv preprint arXiv:1706.01502, 2017.
  11. Randomized ensembled double q-learning: Learning fast without a model. In International Conference on Learning Representations, 2020.
  12. Expected policy gradients for reinforcement learning. Journal of Machine Learning Research, 21(2020), 2020.
  13. Better exploration with optimistic actor critic. Advances in Neural Information Processing Systems, 32, 2019.
  14. Sample-efficient reinforcement learning by breaking the replay ratio barrier. In The Eleventh International Conference on Learning Representations, 2022.
  15. Optimism in reinforcement learning and kullback-leibler divergence. In 2010 48th Annual Allerton Conference on Communication, Control, and Computing (Allerton), pp.  115–122. IEEE, 2010.
  16. Offline reinforcement learning: Fundamental barriers for value function approximation. In Conference on Learning Theory, pp.  3489–3489. PMLR, 2022.
  17. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pp.  1587–1596. PMLR, 2018.
  18. On upper-confidence bound policies for switching bandit problems. In International Conference on Algorithmic Learning Theory, pp.  174–188. Springer, 2011.
  19. Spectral normalisation for deep reinforcement learning: an optimisation perspective. In International Conference on Machine Learning, pp.  3734–3744. PMLR, 2021.
  20. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905, 2018.
  21. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104, 2023.
  22. Gan ensemble for anomaly detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp.  4090–4097, 2021.
  23. Temporal difference learning for model predictive control. In International Conference on Machine Learning, PMLR, 2022.
  24. Hasselt, H. Double q-learning. Advances in neural information processing systems, 23, 2010.
  25. Dropout q-functions for doubly efficient reinforcement learning. In International Conference on Learning Representations, 2021.
  26. Multilayer feedforward networks are universal approximators. Neural networks, 2(5):359–366, 1989.
  27. When to trust your model: Model-based policy optimization. Advances in Neural Information Processing Systems, 32, 2019.
  28. Prospect theory: An analysis of decision under risk. Econometrica, 47(2):263–292, 1979.
  29. Kakade, S. M. A natural policy gradient. Advances in neural information processing systems, 14, 2001.
  30. On bayesian upper confidence bounds for bandit problems. In Artificial intelligence and statistics, pp.  592–600. PMLR, 2012.
  31. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  32. Kostrikov, I. JAXRL: Implementations of Reinforcement Learning algorithms in JAX, 10 2021. URL https://github.com/ikostrikov/jaxrl.
  33. A simple weight decay can improve generalization. Advances in neural information processing systems, 4, 1991.
  34. Discor: Corrective feedback in reinforcement learning via distribution correction. Advances in Neural Information Processing Systems, 33:18560–18572, 2020.
  35. Controlling overestimation bias with truncated mixture of continuous distributional quantile critics. In International Conference on Machine Learning, pp.  5556–5566. PMLR, 2020.
  36. Reinforcement learning with augmented data. Advances in neural information processing systems, 33:19884–19895, 2020.
  37. Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning. In International Conference on Machine Learning, pp.  6131–6141. PMLR, 2021.
  38. Efficient deep reinforcement learning requires regulating overfitting. In The Eleventh International Conference on Learning Representations, 2022.
  39. Regularization matters in policy optimization-an empirical study on continuous control. In International Conference on Learning Representations, 2020.
  40. Understanding plasticity in neural networks. arXiv preprint arXiv:2303.01486, 2023.
  41. Human-level control through deep reinforcement learning. nature, 518(7540):529–533, 2015.
  42. Tactical optimism and pessimism for deep reinforcement learning. Advances in Neural Information Processing Systems, 34:12849–12863, 2021.
  43. A unifying view of optimism in episodic reinforcement learning. Advances in Neural Information Processing Systems, 33:1392–1403, 2020.
  44. The primacy bias in deep reinforcement learning. In International conference on machine learning, pp.  16828–16847. PMLR, 2022.
  45. Deep exploration via bootstrapped dqn. Advances in neural information processing systems, 29, 2016.
  46. Randomized prior functions for deep reinforcement learning. Advances in Neural Information Processing Systems, 31, 2018.
  47. Self-supervised exploration via disagreement. In International conference on machine learning, pp.  5062–5071. PMLR, 2019.
  48. Natural actor-critic. Neurocomputing, 71(7-9):1180–1190, 2008.
  49. A first course in real analysis. Springer Science & Business Media, 2012.
  50. Puterman, M. L. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
  51. Learning representations by back-propagating errors. nature, 323(6088):533–536, 1986.
  52. Decoupled reinforcement learning to stabilise intrinsically-motivated exploration. In Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems, pp.  1146–1154, 2022.
  53. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604–609, 2020.
  54. Trust region policy optimization. In International conference on machine learning, pp.  1889–1897. PMLR, 2015.
  55. Bigger, better, faster: Human-level atari with human-level efficiency. In International Conference on Machine Learning, pp.  30365–30380. PMLR, 2023.
  56. Planning to explore via self-supervised world models. In International Conference on Machine Learning, pp.  8583–8592. PMLR, 2020.
  57. Is bang-bang control all you need? solving continuous control with bernoulli policies. Advances in Neural Information Processing Systems, 34:27209–27221, 2021.
  58. Learning to plan optimistically: Uncertainty-guided deep exploration via latent model ensembles. In Conference on Robot Learning, pp.  1156–1167. PMLR, 2022.
  59. Deterministic policy gradient algorithms. In International conference on machine learning, pp.  387–395. PMLR, 2014.
  60. Mastering the game of go without human knowledge. nature, 550(7676):354–359, 2017.
  61. A walk in the park: Learning to walk in 20 minutes with model-free reinforcement learning. arXiv preprint arXiv:2208.07860, 2022.
  62. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929–1958, 2014.
  63. Stiglitz, J. E. Microeconomics. New York, NY: WW Norton, 1997.
  64. Reinforcement learning: An introduction. MIT press, 2018.
  65. Deepmind control suite. arXiv preprint arXiv:1801.00690, 2018.
  66. Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
  67. A theoretical and empirical analysis of expected sarsa. In 2009 ieee symposium on adaptive dynamic programming and reinforcement learning, pp.  177–184. IEEE, 2009.
  68. Theory of games and economic behavior, 2nd rev. 1947.
  69. Striving for simplicity and performance in off-policy drl: Output normalization and non-uniform sampling. In International Conference on Machine Learning, pp.  10070–10080. PMLR, 2020a.
  70. Optimism in reinforcement learning with generalized linear function approximation. In International Conference on Learning Representations, 2020b.
  71. The role of coverage in online reinforcement learning. In The Eleventh International Conference on Learning Representations, 2022.
  72. Drm: Mastering visual reinforcement learning through dormant ratio minimization. arXiv preprint arXiv:2310.19668, 2023.
  73. A consensus novelty detection ensemble approach for anomaly detection in activities of daily living. Applied Soft Computing, 83:105613, 2019.
  74. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pp.  1094–1100. PMLR, 2020.
  75. Fast and precise: Adjusting planning horizon with adaptive subgoal search. In The Eleventh International Conference on Learning Representations, 2022.
  76. Offline reinforcement learning with realizability and single-policy concentrability. In Conference on Learning Theory, pp.  2730–2775. PMLR, 2022.
Citations (1)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now