Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
126 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Normality-Guided Distributional Reinforcement Learning for Continuous Control (2208.13125v3)

Published 28 Aug 2022 in cs.LG and cs.AI

Abstract: Learning a predictive model of the mean return, or value function, plays a critical role in many reinforcement learning algorithms. Distributional reinforcement learning (DRL) has been shown to improve performance by modeling the value distribution, not just the mean. We study the value distribution in several continuous control tasks and find that the learned value distribution is empirical quite close to normal. We design a method that exploits this property, employ variances predicted from a variance network, along with returns, to analytically compute target quantile bars representing a normal for our distributional value function. In addition, we propose a policy update strategy based on the correctness as measured by structural characteristics of the value distribution not present in the standard value function. The approach we outline is compatible with many DRL structures. We use two representative on-policy algorithms, PPO and TRPO, as testbeds. Our method yields statistically significant improvements in 10 out of 16 continuous task settings, while utilizing a reduced number of weights and achieving faster training time compared to an ensemble-based method for quantifying value distribution uncertainty.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (37)
  1. Joshua Achiam. Spinning Up in Deep Reinforcement Learning, 2018. OpenAI.
  2. Sparse quantile huber regression for efficient and robust estimation. ArXiv, abs/1402.4624, 2014.
  3. Distributed distributional deterministic policy gradients. CoRR, abs/1804.08617, 2018.
  4. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253–279, jun 2013.
  5. Unifying count-based exploration and intrinsic motivation. CoRR, abs/1606.01868, 2016.
  6. A distributional perspective on reinforcement learning. CoRR, abs/1707.06887, 2017.
  7. The cramer distance as a solution to biased wasserstein gradients. CoRR, abs/1705.10743, 2017.
  8. Openai gym. CoRR, abs/1606.01540, 2016.
  9. Exploration by random network distillation. CoRR, abs/1810.12894, 2018.
  10. Estimating risk and uncertainty in deep reinforcement learning, 2019.
  11. Distributional reinforcement learning with quantile regression. CoRR, abs/1710.10044, 2017.
  12. Addressing function approximation error in actor-critic methods. CoRR, abs/1802.09477, 2018.
  13. Q-prop: Sample-efficient policy gradient with an off-policy critic. arXiv preprint arXiv:1611.02247, 2016.
  14. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. CoRR, abs/1801.01290, 2018.
  15. Stephen C. Hora. Aleatory and epistemic uncertainty in probability elicitation with an example from hazardous waste management. Reliability Engineering & System Safety, 54(2):217–223, 1996. Treatment of Aleatory and Epistemic Uncertainty.
  16. Peter J. Huber. Robust Estimation of a Location Parameter. The Annals of Mathematical Statistics, 35(1):73 – 101, 1964.
  17. Galin L. Jones. On the Markov chain central limit theorem. Probability Surveys, 1(none):299 – 320, 2004.
  18. What uncertainties do we need in bayesian deep learning for computer vision? CoRR, abs/1703.04977, 2017.
  19. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2014.
  20. Aleatory or epistemic? does it matter? Structural Safety, 31(2):105–112, 2009. Risk Acceptance and Risk Communication.
  21. Simple and scalable predictive uncertainty estimation using deep ensembles, 2016.
  22. SUNRISE: A simple unified framework for ensemble learning in deep reinforcement learning. CoRR, abs/2007.04938, 2020.
  23. Sample efficient deep reinforcement learning via uncertainty estimation. CoRR, abs/2201.01666, 2022.
  24. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pages 1928–1937. PMLR, 2016.
  25. Efficient exploration with double uncertain value networks. CoRR, abs/1711.10789, 2017.
  26. Estimating the mean and variance of the target probability distribution. In Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN’94), volume 1, pages 55–60 vol.1, 1994.
  27. Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
  28. Trust region policy optimization. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 1889–1897, Lille, France, 07–09 Jul 2015. PMLR.
  29. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438, 2015.
  30. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017.
  31. Sample-based distributional policy gradient. CoRR, abs/2001.02652, 2020.
  32. A theoretical analysis of model-based interval estimation. In ICML, pages 856–863, 2005.
  33. Reinforcement learning: An introduction. MIT press, 2018.
  34. Philip Thomas. Bias in natural actor-critic algorithms. In International conference on machine learning, pages 441–448. PMLR, 2014.
  35. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026–5033, 2012.
  36. Q-learning. Machine learning, 8(3):279–292, 1992.
  37. Implicit distributional reinforcement learning. CoRR, abs/2007.06159, 2020.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now