Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
157 tokens/sec
GPT-4o
43 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Mollification Effects of Policy Gradient Methods (2405.17832v1)

Published 28 May 2024 in cs.LG, cs.AI, and cs.RO

Abstract: Policy gradient methods have enabled deep reinforcement learning (RL) to approach challenging continuous control problems, even when the underlying systems involve highly nonlinear dynamics that generate complex non-smooth optimization landscapes. We develop a rigorous framework for understanding how policy gradient methods mollify non-smooth optimization landscapes to enable effective policy search, as well as the downside of it: while making the objective function smoother and easier to optimize, the stochastic objective deviates further from the original problem. We demonstrate the equivalence between policy gradient methods and solving backward heat equations. Following the ill-posedness of backward heat equations from PDE theory, we present a fundamental challenge to the use of policy gradient under stochasticity. Moreover, we make the connection between this limitation and the uncertainty principle in harmonic analysis to understand the effects of exploration with stochastic policies in RL. We also provide experimental results to illustrate both the positive and negative aspects of mollification effects in practice.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (56)
  1. Stochastic optimization and sparse statistical recovery: An optimal algorithm for high dimensions. arXiv preprint arXiv:1207.4421, 2012.
  2. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research, 22(98):1–76, 2021.
  3. On the linear convergence of policy gradient methods for finite MDPs. International Conference on Artificial Intelligence and Statistics, 2021.
  4. Variable smoothing for convex optimization problems using stochastic gradients. Journal of Scientific Computing, 85(33), 2020.
  5. Openai gym. arXiv preprint arXiv:1606.01540, 2016.
  6. Provably efficient exploration in policy optimization. Proceedings of the 37th International Conference on Machine Learning, pp.  1283–1294, 2020.
  7. Neural Lyapunov control. Advances in Neural Information Processing Systems, 32:3245–3254, 2019.
  8. Faster gradient-free algorithms for nonsmooth nonconvex stochastic optimization. arXiv preprint arXiv:2301.06428, 2023.
  9. Evans, L. C. Partial Differential Equations. American Mathematical Society, 2010.
  10. Global convergence of policy gradient methods for the linear quadratic regulator. Proceedings of the 35th International Conference on Machine Learning, pp.  1467–1476, 2018.
  11. Addressing function approximation error in actor-critic methods. Proceedings of the 35th International Conference on Machine Learning, pp.  1587–1596, 2018.
  12. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. arXiv preprint arXiv:1610.00633, 2016.
  13. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. Proceedings of the 35th International Conference on Machine Learning, pp.  1861–1870, 2018a.
  14. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905, 2018b.
  15. Deep Q-learning from demonstrations. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), 2018.
  16. Höllig, K. Existence of infinitely many solutions for a forward backward heat equation. Transactions of the American Mathematical Society, 278(1), 1983.
  17. Is q-learning provably efficient? Advances in Neural Information Processing Systems, 31, 2018.
  18. Provably efficient reinforcement learning with linear function approximation. Proceedings of Thirty Third Conference on Learning Theory, pp.  2137–2143, 2020.
  19. Kabanikhin, S. I. Definitions and examples of inverse and ill-posed problems. J. Inv. Ill-Posed Problems, 16:317–357, 2008.
  20. Kakade, S. M. A natural policy gradient. Advances in Neural Information Processing Systems 14, pp. 1531–1538, 2001.
  21. Human Motion Control of Quadrupedal Robots using Deep Reinforcement Learning. Proceedings of Robotics: Science and Systems, 2022.
  22. Actor-critic algorithms. Advances in Neural Information Processing Systems, 12, 1999.
  23. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015.
  24. Gradient-free methods for deterministic and stochastic nonsmooth nonconvex optimization. arXiv preprint arXiv:2209.05045, 2022.
  25. Gradients are not all you need. 2021.
  26. Parmas, P. Total stochastic gradient algorithms and applications in reinforcement learning. NeurIPS, 2018.
  27. A unified view of likelihood ratio and reparameterization gradients. International Conference on Artificial Intelligence and Statistics, pp.  4078–4086, 2021.
  28. Pipps: Flexible model-based policy search robust to the curse of chaos. Proceedings of the 35th International Conference on Machine Learning, pp.  4065–4074, 2018.
  29. Natural actor-critic. Neurocomputing, 71(7):1180–1190, 2008.
  30. Extrema and nowhere differentiable functions. The Rocky Mountain Journal of Mathematics, 16(4):661–668, 1986.
  31. An introduction to partial differential equations. Springer, 1993.
  32. Rudin, W. Functional Analysis. McGraw-Hill, Inc., 1991.
  33. Universal value function approximators. Proceedings of the 32nd International Conference on Machine Learning, pp.  1312–1320, 2015.
  34. Trust region policy optimization. Proceedings of the 32nd International Conference on Machine Learning, pp.  1889–1897, 2015a.
  35. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438, 2015b.
  36. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
  37. Shamir, O. An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. arXiv preprint arXiv:1507.08752, 2015.
  38. Reachability-guided sampling for planning under differential constraints. IEEE International Conference on Robotics and Automation, pp. 2859–2865, 2009.
  39. Deterministic policy gradient algorithms. Proceedings of the 31st International Conference on Machine Learning, pp.  387–395, 2014.
  40. Fourier Analysis: An Introduction. Princeton University Press, 2003.
  41. Do differentiable simulators give better policy gradients? Proceedings of the 39th International Conference on Machine Learning, 162:20668–20696, 2022.
  42. Reinforcement Learning: an Introduction. MIT Press, 1998.
  43. Policy gradient methods for reinforcement learning with function approximation. Advances in Neural Information Processing Systems 12, pp. 1057–1063, 1999.
  44. Analysis of the optimization landscape of linear quadratic gaussian (LQG) control. Proceedings of the 3rd Conference on Learning for Dynamics and Control, pp.  599–610, 2021.
  45. Tedrake, R. Underactuated Robotics: Algorithms for Walking, Running, Swimming, Flying, and Manipulation (Course Notes for MIT 6.832).
  46. Deep reinforcement learning with double Q-learning. Proceedings of the AAAI Conference on Artificial Intelligence, 30(1), 2016.
  47. Fractal landscapes in policy optimization. arXiv preprint arXiv:2310.15418, 2023.
  48. Stochastic zeroth-order optimization in high dimensions. pp.  1356–1365, 2018.
  49. Optimism in reinforcement learning with generalized linear function approximation. arXiv preprint arXiv:1912.04136, 2019.
  50. Q-learning. Machine Learning, 8:279–292, 1992.
  51. Natural evolution strategies. Journal of Machine Learning Research, 15:949–980, 2014.
  52. Williams, R. J. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8:229–256, 1992.
  53. Xiao, L. On the convergence rates of policy gradient methods. arXiv preprint arXiv:2201.07443, 2022.
  54. Connected superlevel set in (deep) reinforcement learning and its application to minimax theorems. arXiv preprint arXiv:2303.12981, 2023.
  55. Policy optimization for ℋ2subscriptℋ2\mathcal{H}_{2}caligraphic_H start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT linear control with ℋ∞subscriptℋ\mathcal{H}_{\infty}caligraphic_H start_POSTSUBSCRIPT ∞ end_POSTSUBSCRIPT robustness guarantee: Implicit regularization and global convergence. SIAM Journal on Control and Optimization, 59(6):4081–4109, 2021.
  56. Adaptive barrier smoothing for first-order policy gradient with contact dynamics. Proceedings of Machine Learning Research, pp.  41219–41243, 2023.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now