Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
153 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Parametric PDE Control with Deep Reinforcement Learning and Differentiable L0-Sparse Polynomial Policies (2403.15267v2)

Published 22 Mar 2024 in cs.LG

Abstract: Optimal control of parametric partial differential equations (PDEs) is crucial in many applications in engineering and science. In recent years, the progress in scientific machine learning has opened up new frontiers for the control of parametric PDEs. In particular, deep reinforcement learning (DRL) has the potential to solve high-dimensional and complex control problems in a large variety of applications. Most DRL methods rely on deep neural network (DNN) control policies. However, for many dynamical systems, DNN-based control policies tend to be over-parametrized, which means they need large amounts of training data, show limited robustness, and lack interpretability. In this work, we leverage dictionary learning and differentiable L$_0$ regularization to learn sparse, robust, and interpretable control policies for parametric PDEs. Our sparse policy architecture is agnostic to the DRL method and can be used in different policy-gradient and actor-critic DRL algorithms without changing their policy-optimization procedure. We test our approach on the challenging tasks of controlling parametric Kuramoto-Sivashinsky and convection-diffusion-reaction PDEs. We show that our method (1) outperforms baseline DNN-based DRL policies, (2) allows for the derivation of interpretable equations of the learned optimal control laws, and (3) generalizes to unseen parameters of the PDE without retraining the policies.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (78)
  1. Optimal control of partial differential equations. Springer, 2021.
  2. Optimal control. John Wiley & Sons, 2012.
  3. Robert F Stengel. Optimal control and estimation. Courier Corporation, 1994.
  4. Optimal control: an introduction to the theory and its applications. Courier Corporation, 2013.
  5. Donald E Kirk. Optimal control theory: an introduction. Courier Corporation, 2004.
  6. Arthur Earl Bryson. Applied optimal control: optimization, estimation and control. CRC Press, 1975.
  7. Fredi Tröltzsch. Optimal control of partial differential equations: theory, methods, and applications, volume 112. American Mathematical Soc., 2010.
  8. Jacques Louis Lions. Optimal control of systems governed by partial differential equations, volume 170. Springer, 1971.
  9. Sparse identification of nonlinear dynamics for model predictive control in the low-data limit. Proceedings of the Royal Society of London A, 474(2219), 2018.
  10. An optimal control approach to robust control of robot manipulators. IEEE Transactions on robotics and automation, 14(1):69–77, 1998.
  11. A strongly-coupled immersed-boundary formulation for thin elastic structures. Journal of Computational Physics, 336:401–411, 2017.
  12. Data-driven unsteady aeroelastic modeling for control. AIAA Journal, 61(2):780–792, 2023.
  13. W Fred Ramirez. Application of optimal control theory to enhanced oil recovery. Elsevier, 1987.
  14. Integrating process design and control: An application of optimal control to chemical processes. Chemical Engineering and Processing: Process Intensification, 47(11):2004–2018, 2008.
  15. George W Swan et al. Applications of optimal control theory in biomedicine. M. Dekker New York, 1984.
  16. Lennart Ljung. System identification. In Signal analysis and prediction, pages 163–173. Springer, 1998.
  17. Reinforcement learning: An introduction. MIT press, 2018.
  18. Deep reinforcement learning: A brief survey. IEEE Signal Processing Magazine, 34(6):26–38, 2017.
  19. Yuxi Li. Deep reinforcement learning: An overview. arXiv preprint arXiv:1701.07274, 2017.
  20. An introduction to deep reinforcement learning. Foundations and Trends® in Machine Learning, 11(3-4):219–354, 2018.
  21. Human-level control through deep reinforcement learning. nature, 518(7540):529–533, 2015.
  22. Mastering complex control in moba games with deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 6672–6679, 2020.
  23. A survey of deep reinforcement learning in video games. arXiv preprint arXiv:1912.10944, 2019.
  24. Playing fps games with deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
  25. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
  26. Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
  27. Dueling network architectures for deep reinforcement learning. In International conference on machine learning, pages 1995–2003. PMLR, 2016.
  28. Long-Ji Lin. Reinforcement learning for robots using neural networks. Carnegie Mellon University, 1992.
  29. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 32(11):1238–1274, 2013.
  30. Survey of model-based reinforcement learning: Applications on robotics. Journal of Intelligent & Robotic Systems, 86(2):153–173, 2017.
  31. On reward shaping for mobile robot navigation: A reinforcement learning and slam based approach. arXiv preprint arXiv:2002.04109, 2020.
  32. Towards vision-based deep reinforcement learning for robotic motion control. In Australasian Conference on Robotics and Automation 2015. Australian Robotics and Automation Association (ARAA), 2015.
  33. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In 2017 IEEE international conference on robotics and automation (ICRA), pages 3389–3396. IEEE, 2017.
  34. Sim-to-real transfer in deep reinforcement learning for robotics: a survey. In 2020 IEEE symposium series on computational intelligence (SSCI), pages 737–744. IEEE, 2020.
  35. Low dimensional state representation learning with robotics priors in continuous action spaces. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 190–197. IEEE, 2021.
  36. Scientific multi-agent reinforcement learning for wall-models of turbulent flows. Nature Communications, 13(1):1443, 2022.
  37. Controlling rayleigh–benard convection via reinforcement learning. Journal of Turbulence, 21(9-10):585–605, 2020.
  38. Optimal control of point-to-point navigation in turbulent time dependent flows using reinforcement learning. Springer, 2020.
  39. Deep reinforcement learning in fluid mechanics: A promising method for both active flow control and shape optimization. Journal of Hydrodynamics, 32:234–246, 2020.
  40. Recent advances in applying deep reinforcement learning for flow control: Perspectives and future directions. Physics of Fluids, 35(3), 2023.
  41. Reinforcement learning for bluff body active flow control in experiments and simulations. Proceedings of the National Academy of Sciences, 117(42):26091–26098, 2020.
  42. Accelerating deep reinforcement learning strategies of flow control through a multi-environment approach. Physics of Fluids, 31(9), 2019.
  43. Active flow control for bluff body drag reduction using reinforcement learning with partial measurements. Journal of Fluid Mechanics, 981:A17, 2024.
  44. Distributed control of partial differential equations using convolutional reinforcement learning. Physica D: Nonlinear Phenomena, page 134096, 2024.
  45. SINDy-RL: Interpretable and efficient model-based reinforcement learning. arXiv preprint arXiv:2403.09110, 2024.
  46. State representation learning for control: An overview. Neural Networks, 108:379–392, 2018.
  47. Unsupervised representation learning in deep reinforcement learning: A review. arXiv preprint arXiv:2208.14226, 2022.
  48. Imitation learning: A survey of learning methods. ACM Comput. Surv., 50(2), 2017.
  49. Behavioral cloning from observation. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pages 4950–4957, 2018.
  50. Exploring the limitations of behavior cloning for autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9329–9338, 2019.
  51. Transfer learning. In Handbook of research on machine learning applications and trends: algorithms, methods, and techniques, pages 242–264. IGI global, 2010.
  52. A survey of transfer learning. Journal of Big data, 3(1):1–40, 2016.
  53. Meta-learning in neural networks: A survey. IEEE transactions on pattern analysis and machine intelligence, 44(9):5149–5169, 2021.
  54. A perspective view and survey of meta-learning. Artificial intelligence review, 18:77–95, 2002.
  55. Explainable artificial intelligence: A survey. In 2018 41st International convention on information and communication technology, electronics and microelectronics (MIPRO), pages 0210–0215. IEEE, 2018.
  56. Learning sparse neural networks through l_0 regularization. In International Conference on Learning Representations, 2018.
  57. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149, 2015.
  58. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107–115, 2021.
  59. Variational dropout sparsifies deep neural networks. In International Conference on Machine Learning, pages 2498–2507. PMLR, 2017.
  60. Soft weight-sharing for neural network compression. In International Conference on Learning Representations, 2016.
  61. Online dictionary learning for sparse coding. In Proceedings of the 26th annual international conference on machine learning, pages 689–696, 2009.
  62. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the national academy of sciences, 113(15):3932–3937, 2016.
  63. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  64. Q-learning. Machine learning, 8:279–292, 1992.
  65. Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8:229–256, 1992.
  66. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015.
  67. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
  68. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861–1870. PMLR, 2018.
  69. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587–1596. PMLR, 2018.
  70. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint arXiv:1611.00712, 2016.
  71. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144, 2016.
  72. A unified framework for sparse relaxed regularized regression: Sr3. IEEE Access, 7:1404–1423, 2018.
  73. A unified sparse optimization framework to learn parsimonious physics-informed models from data. IEEE Access, 8:169259–169271, 2020.
  74. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
  75. Hypersindy: Deep generative modeling of nonlinear stochastic governing equations. arXiv preprint arXiv:2310.04832, 2023.
  76. Nikolai A Kudryashov. Exact solutions of the generalized kuramoto-sivashinsky equation. Physics Letters A, 147(5-6):287–291, 1990.
  77. Controlgym: Large-scale safety-critical control environments for benchmarking reinforcement learning algorithms. arXiv preprint arXiv:2311.18736, 2023.
  78. Provably efficient rl with rich observations via latent state decoding. In International Conference on Machine Learning, pages 1665–1674. PMLR, 2019.
Citations (6)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets